pasUnity 4.31.0.0 RTM is now generally available.

This release features enhancements to concurrency in large environments, some usage enhancements to the history and exception views, and an all-new data file processing wizard.

The concurrency change is that the locking mechanism used to coordinate I/O operations for Data Files is migrated off the agents and into the SQL Server database.  Now application-wide transactional locking ensures that all new Data Files added to the system are processed serially such that no DropBox impeller criteria is ever missed and that concurrency issues (such as deadlocks) do not occur.  Further, the validation of Matrix rules such as storage limits, multiple upload rules, and naming conventions undergo validation BEFORE the file is loaded into permanent storage to prevent having to roll changes back.

The exception and job history views now have the ability so configure the maximum number of entries to show the end user.  Who would have guessed that 500 entries wouldn't be enough for some people?  Me.  But now that I know better the ability existing to asks for millions of entries at once.  Additionally, the exception views now allow you to provide filter strings to quickly locate a subset of exceptions.

Necessity is the mother of invention. –Old English Proverb

So one of our customers had a problem.  They had an entire month’s worth of integrations to a BI repository that needed to be reversed and then reloaded again using an updated mapping sheet to transform the input.  Not a big deal right?  Just delete the output files from previous operations and start clicking the reprocess button for each source data file on the matrix right?  Well the source data consisted of over 24 thousand files.  Now that is an awful lot of double-clicking.  This prompted us to build an all-new feature for pasUnity: the Data File Processing Wizard. 

The wizard is available to administrative users by right-clicking on a Matrix object in the pasUnity UI and selecting “Process Data Files…” which launches the wizard.

The first thing you are asked to do is select the DropBoxes you want to search in.  To make this easy we even put some mass-toggle options in there for you as shown below.
  


Once you have made your DropBox selections move next.  Now you get to specify the search criteria for locating the Data Files you want to act on.  You can use wildcards in the text boxes and choose to as much or as little criteria as needed. 



Pressing next shows you the search results displaying all Data Files that meet your criteria.  Select Data Files individually or use the mass-toggle option to select at least one Data File and press Next.



Finally, you get to choose the action to perform.  You currently have three choices.  You can choose to delete all the Data Files which will instantaneously wipe them out and also remove any work in the queue related to the files.  Alternatively, you can choose to reprocess the files.  When you reprocess the Data Files are processed as though they were being freshly added to the system by detecting all DropBox impellers that meet activation criteria and queueing up the jobs.  If you do not want to be quite so heavy handed you can choose from a list of specific jobs as shown below.



After making your action selection and providing any supplemental configuration options, click Next and the action is immediately executed.  Be careful – there is NO undo button here.

So back to the story.  In roughly one minute, we were able to punch in the criteria to remove all the intermediate and final output files from the original integrations and wipe them from the database.  In a couple minutes more, we ran through the wizard a couple more times and marked all of the original source files to be re-processed.  This resulted in queuing up nearly 19K jobs for processing which in turn queued up another 5K jobs to process the results from the previous jobs.  Now all of this work executed on a single pasUnity agent with the worker threads set to 48 in a period of less than 3 hours.  Yeah: over 24K files processed through pasUnity and pasTransfer (including mapping) and posted through to the destination system in less than 3 hours on a single machine – with no errors.