Execute BizTalk Map in pipeline – Could be savior when transforming huge messages

Posted on Updated on

Transformation is one of the key elements in integration and BizTalk facilitates this in way which is one of the best.

Transformations in BizTalk can be done within the Orchestrations , or at the port level by specifying inbound or outbound maps.

Transformations can also be executed on pipelines so as to avoid persisting messages to database if the source is huge.

For example, you are receiving a big message say 100 MB which contains all the records extracted but for your business process within BizTalk we need only a subset of that, that meets some condition. One way of achieving this is by applying the filtering map at the port level and then feeding the business process orchestration with the smaller set. However there is limitation of one map that can be executed on port and for such scenario where we need to execute multiple maps on source the drawback here would be that whole message say 100-150 MB will be persisted to message box before further transforms applies and this becomes overhead if there are many such processes.

To overcome this, executing the map at the pipeline level before the persistence goes a long way in improving performance.

Pipeline

For this we can define a pipeline component with below properties.

PipelineProps

MapFQN is the fully qualified name for the map to be executed.

(The same can be extended to include xsl transformations as well. Say a path to XSLT file that needs to be executed for transformation.)

ExecuteMap

MapVirtual

Virtual stream takes care of disk offloading during transformation for big messages thus greatly reducing the server memory load.

 

Advertisements

4 thoughts on “Execute BizTalk Map in pipeline – Could be savior when transforming huge messages

    fernandodosanjos said:
    August 24, 2016 at 8:59 pm

    I was of the impression that messages was saved
    In messagebox after regular receive port map execution not before?

      Ritu Raj responded:
      August 25, 2016 at 3:44 am

      Hi Fernando , For a one way receive port, thats true.
      Pipeline map execution comes in handy for cases where you need to execute multiple maps. For a port level map its restricted to one map.

    Maheshkumar Tiwari said:
    August 24, 2016 at 4:04 pm

    Something new to read … But am wondering what happens if there is failure while transforming or in case where some debugging is to be done on the original message received… I hope my point is clear.

      Ritu Raj responded:
      August 24, 2016 at 5:57 pm

      Hi Mahesh, I have used this pattern for a project where a huge csv file had to be handled and mapped for further processing. A 100 MB csv file was getting enormous in size after xml conversion for the orchestration to subscribe and do transformations. I used the above approach to filter out needed message and the application behaviour was consistent and low memory impact on the server.

      For any failures in the pipeline, the behaviour would be same as any other failure of a pipeline and errors can be subscribed. For debugging the failed message can be inspected. For any custom behaviour , I would prefer handling the transformation error in pipeline and creating a failure message with details and submitting to message box for appropriate subscription.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s