Aggregation Pattern for Large Batches

Posted on Updated on

PS: This is continuation of this great blog post, and aims to address few drawbacks

============================================================================

One of the many frequent requirements in integration is delivering aggregated data to systems for further processing. BizTalk facilitates this and there are several aggregation pattern in BizTalk.

But for large message sets many of these patterns are insufficient. During a recent project we had a similar requirement. I came across this blog post that solved the problem for us. There were few tweaks that we introduced because there were few problems specifically the delay before the batch gets written was something not safe to assume for us as the batches were big in size and load on the server was high. So from a performance and reliability point of view we had to adjust it.

We also tweaked how the header and trailer are written to make it more generic.

A guaranteed delivery and write of the batch message was achieved by using the DeliveryNotification in BizTalk.

AggregateCapture

 

Delivery failure notification took care of writing the complete batch to file so this enabled removing of the delay while writing the batches and thus making the aggregation reliable.

If you prefer to use static port then the file adapter should have Append to existing file property selected and in case of dynamic port

AggregatePropertyCapture

Instead of writing Header and Footer while writing the file, we tweaked it to write header and footer while reading the file. While reading the file in the orchestrations we can construct the final message as below.

AggregateHeaderTrailerCapture

This can also be implemented at a pipeline level to make it more generic. Once the file is written to the disk , a receive location can be used to read the file and then send it to destination systems.

On the receive pipeline define.

AggregatePipeCapture

With this implementaion it was posible to process 2800-3000 batches quickly and with low memory usage.

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s