Batch processes integrate all of the data matching a given criteria into the target system in one large process. Real-time processes monitor the source system for new transactions and respond immediately to a new transaction by integrating the transaction with the target system transaction-by-transaction, in real-time. Batch processes take a great deal of processing power and a long time to complete, since they must processes all records that meet the integration criteria at once. Speed ratings (transactions per second for instance) are critical for batch processes because they can run for a long time to process all records. You may need to ensure that the integration completes in an acceptable amount of time each time it is run. Real-time processes monitor and integrate data one transaction at a time as they appear. This means that teal-time integration processes are always ‘caught-up’, so the transactions per second capabilities are of much less concern for these processes.
Batch versus Real-Time – What is the best choice?
There is no methodology that is better than another in all cases. In some cases, it makes sense to automate integration via batch processes, and in other cases real-time integration makes more sense. A mature and capable integration solution will allow for each integration map to be integrated as you see fit. A typical implementation will see a mixture of maps automated as batch processes and real-time processes.
Batch is best when …
Batch should be considered when:
The data being integrated is not time-sensitive
If the data you are integrating has little impact from being hours or days old, then batch integration could be a good choice.
You have limited integration between systems and you need to be sure that all changes from many areas are kept in synch with as few processes as possible.
If you are unable or unwilling to create data maps for all of the individual elements requiring integration, a batch processes that integrates many fields could save time and get the integration process up and running sooner.
You have to integrate a large number of data elements, but not very many transactions (records).
One of the pitfalls of a batch integration is the amount of time it takes to process large source records sets. if the number of transactions occurring is minimal, then batch integration could be a good choice for integration.
The data be in integrated is closely coupled with other data that is being updated on a batch basis.
If you are integrating data that is closely coupled to data that is being updated via a batch process, then it only makes sense that the closely coupled data should be updated via a batch process as well. For instance, if you are using an ERP system and integrating forecast orders (that are updated via a batch process) to another system, there may be no advantage to monitor the forecast orders in real-time since they are only updated with each ERP batch run. In this case, it might be more efficient to set up a batch integration that is triggered just after the forecast orders are updated.
You don’t mind a little manual intervention
Batch processes can be automated just like real-time processes can be, but batch processes have a catch. If a special situation arises where a refresh of the data is required that falls out of the normal batch processing times, then the batch synchronization will need to be triggered manually. This can be a big deal if the person with all of the system smarts is not available or if the batch takes an excessively long time to complete. This situation doesn’t come up every day, but it always comes up eventually when dealing with batch processes.
You have the necessary power to run large batch processes
Batch processes can take a long time to run and take up a lot of resources. Since these processes usually have a time window in which they need to complete, adding more processes can make for a time crunch or a processing power crunch. Before choosing to integrate systems using batch processes, be sure that you have excess capacity and/or time to allow the integration to complete in an acceptable amount of time.
Real-Time is best when …
Real-time integration should be considered when:
The data being integrated is time-sensitive
If the data you are integrating has an impact if it is hours old, then it might be good to choose real-time integration to ensure that the data is always as current as possible.
You have the development expertise and time required to create integration maps that describe exactly what needs to be integrated, without resorting to integrating large collections of data when only specific data within those sets is required.
It’s ideal to be able to integrate only the data that is required to be integrated. Sometimes a shortage of development time or expertise leads to more data being integrated than is necessary. If you have the time and skill to create exacting integration maps, then you can create a greater number of real-time integration processes rather than a smaller number of batch processes.
You have to integrate many transactions (records).
If you are integrating systems with high transactional throughput, then real-time integration can be a better choice because it is always caught-up with processing. This averts the problem of excessively long batch runs that can hog machine resources and interfere with other processes.
The data be in integrated is loosely coupled with other data that is being updated on a batch basis.
Most environments have a mix of data that is updated in real-time and data that is updated via batch processes. If the data you are integrating is not tied to a batch processes, then you are free to choose real-time integration to integrate it with other systems as efficiently as possible.
You want to avoid manual intervention
Real-time processes are always caught-up, so there is rarely a need for manual intervention to get them caught-up or synchronized with other activities. Real-time integrations tend to “just work” and require the least amount of manual attention.
You do not have the necessary power to run large batch processes
Batch processes can take a long time to run and take up a lot of resources. Since these processes usually have a time window in which they need to complete, adding more processes can make for a time crunch or a processing power crunch. If you do not have the excess capacity and/or time to allow a batch integration to complete in an acceptable amount of time then you should choose real-time integration. Real-time integration is much less processor intensive, and does not require a time window to complete.
A Note on THE LINK®
THE LINK allows all maps to be deployed as either a batch process or a real-time process. There is no change in the map itself. This is simply a deploy-time decision. Batch maps can be scheduled. Real-time maps can be associated with up-time and down-time schedules so that they do not run at inappropriate times