Batch processing – the importance of wash, rinse, repeat.

By Winshuttle Staff Blogger on Jul 8, 2013

small car

Photo by: Roger Wolfsburg [CC Attribution 2.0]

Batch processing, or batching, is a scaled-down version of the idea of the assembly line. Activity is aggregated in groups instead of in a continuous stream of progressive assembly. Progressive assembly is pretty familiar to anyone who has or currently works in the manufacturing industry. Conceptually the great economist Adam Smith discussed the whole idea of continuous work and the division of labor in the manufacture of pins in his work The Wealth of Nations. Though online transaction processing seems to have moved largely away from the concept of batch processing, in the world of finance and even in IT, in the back office, batch processing is very much alive.

A year or two ago, I was involved in a ‘show and tell’ regarding how Winshuttle works with SAP and how some of the characteristics of the way that it works makes SAP transactions more accessible and usable to new and novice SAP users. The audience was clearly very intelligent, eager and interested to learn more, but from within their midst one individual commented, well, ‘it’s just batch processing, there is no real-time interaction with the ERP system’. His comment was correct; however, his dismissal of the capabilities presented to him revealed his ignorance of the importance of something as seemingly unimpressive as batching work.

The examples demonstrated showed that a given user can stage their data in advance in Microsoft Excel, Access or even Google spreadsheets. What was not elaborated was the fact that in reality batch processing and the need for batch processing in the back office of many business activities is crucial to ensuring data fidelity, as well as good data governance and process cadence.

Consider the fact that a shopping cart in an online shopping system may involve an element of batch processing – the pricing, the catalog, etc. In fact, until the point at which you hand over your credit card number or close out the transaction with a payment processor, your order is being batch processed. The assumption that because you can have an interactive experience with an online shopping system, means that the process is devoid of batch processing elements is a false one. The only real commercial exceptions to this might be your interaction with your bank for transfers and withdrawals, an auction site, ticketing, reservations and brokerage. There are possibly many others, but in reality most accounting activities and even selling activities are recorded and expedited using batch processing methods.

The reason batching is so ubiquitous and probably will continue to be so, is because of the prevailing value inherent in being able to collate data and review it before “making it so” is your core system of record. The value adding step here is the ‘review’ part. Although review is often not considered an important aspect of batch processing, it is the step that today has probably the greatest significance.

Gartner states that 70% of IT processes alone are still performed in batch rather than real-time, demonstrating that workload automation is a key component in delivering critical business services in IT. Yes, even for IT which holds itself up as an icon for accessibility of real-time information, batch processing needs to be supported. The reason for batch processing in the realm of IT is largely due to the desire to run activity in off peak system usage cycles such as syncing data from the ERP system to the data marts or data warehouse, but it also has value when you want to copy data needed to synchronize stable and disparate system.

In the world of SAP, synchronization and summarization are largely done by running reports and these are often run as batch jobs in off peak hours. In the world of inventory handling and management, the pick-list for example is nothing more than aggregation of items to be picked from the warehouse, which may or may not relate to a single or multiple orders but which has the most efficiency for item selection. Continuous pick list generation may be something interesting when you have free pickers milling about waiting for work but in most optimized operations the pickers are constantly working on picking inventory in the most efficient way possible before packing and shipping. Often the efficiency only flows from batching or waves which are mini batches.

No one would deny that batching can also create delays, because activities stop as you switch to a different batch of work; however, with some intelligent segregation of duties and expansion of application accessibility delays can be dramatically reduced. As zapthink’s Ron Schmelzer put it “Batch processes remain essential for one key reason: it is simply not efficient to regenerate a complete forecast or business plan every time the business processes a single event such as an incoming customer order”. So even in the world of a service oriented architecture, batch processing has its place.

Consider the end-to-end process of account journaling. In the absence of a fully integrated sub ledger system with your core ERP ledger system, you need to entertain the fact that you will periodically create manual journals. Creating journals in a constant flow may not always work with your legacy sub ledger system. For example, transactions may span several time intervals such as days or weeks, but come the end of the period, irrespective of whether you rely on flash reporting or not, you have to get your core accounting system aligned with all those subledger systems and the best way is to batch the activity up, ensure there are no discrepancies and simply make it so. The fact that this is a repeatable process, a stable one and one that happens with regularity doesn’t detract from the overall value of batch it simply makes the end result more consistent, predictable and reliable.

You can read more on how Winshuttle helps hundreds of companies interact with their core ERP using batched methods and even real time interactive alternatives to the traditional screens of ERP in finance by simply taking a look at

Additional reading





About the author

Staff Blogger

The Winshuttle blog is written by professional thought leaders who are dedicated to providing content on a variety of topics, including industry news, best practices, software updates, continued education, tips and techniques, and much more.

Questions or comments about this article?

Tweet @Winshuttle to continue the conversation!