Aggregation Pattern for Large Batches

Posted on Updated on

PS: This is continuation of this great blog post, and aims to address few drawbacks


One of the many frequent requirements in integration is delivering aggregated data to systems for further processing. BizTalk facilitates this and there are several aggregation pattern in BizTalk.

But for large message sets many of these patterns are insufficient. During a recent project we had a similar requirement. I came across this blog post that solved the problem for us. There were few tweaks that we introduced because there were few problems specifically the delay before the batch gets written was something not safe to assume for us as the batches were big in size and load on the server was high. So from a performance and reliability point of view we had to adjust it.

We also tweaked how the header and trailer are written to make it more generic.

A guaranteed delivery and write of the batch message was achieved by using the DeliveryNotification in BizTalk.



Delivery failure notification took care of writing the complete batch to file so this enabled removing of the delay while writing the batches and thus making the aggregation reliable.

If you prefer to use static port then the file adapter should have Append to existing file property selected and in case of dynamic port


Instead of writing Header and Footer while writing the file, we tweaked it to write header and footer while reading the file. While reading the file in the orchestrations we can construct the final message as below.


This can also be implemented at a pipeline level to make it more generic. Once the file is written to the disk , a receive location can be used to read the file and then send it to destination systems.

On the receive pipeline define.


With this implementaion it was posible to process 2800-3000 batches quickly and with low memory usage.


Read documents from SharePoint programmatically using C# (Perform a non deleting BizTalk WSS Recieve)

Posted on Updated on

Documents from SharePoint can be read programmatically using C# WebRequest.

On  a BizTalk front, such need occurs when you want to perform a non deleting receive for a sharepoint document.

Consider a scenario where your business process gets triggered and then within needs to read data from sharepoint document library and use it in the process.

BizTalk by default will delete the document from the library after reading the same. One can argue that we can implement a process within, where in the BizTalk  process will read the document and then put it back to the location through a WSS send port and we are done. This is absolutely fine and works as expected but as you see it has more over heads and more failure point. What if while uploading the document it fails and the nest process never gets to read the data and hence resulting in subsequent failures of the main business process. And above all you still need to have a C# code to activate , deactivate your receive location.

What comes in handy in such business scenario is that we can have the Sharepoint location read from a config, preferably SSO Config and then use C# code to read the data to stream or memory stream and then use the same within our business process.

Using memorystream would also help load it to BizTalk XLAng message. Also we can implement a full BizTalk streaming mechanism here to preform our action using IStreamfactory.  More details on this can be found here .

Using the below code we can read the contents to a stream and then to memorystream. UseDefaultCredentials will run the program under current user account or in BizTalk Orch case, under the BizTalk host instance account. So make sure that the user has sufficient rights to read the document library.

We can also key in specific user under the credentials of the request if desired so. Capture

Install MSI packages (Locally and on Remote computers) using WMI

Posted on Updated on

Installing of packages and msi is a day to day activity in current business scenario. And with growing use of technology and programs it becomes all the more important to have this process efficient, error free, less time consuming. Automating such process becomes all the more important.

Today we will see how we can achieve the same. WMI comes with great feature to leverage it and automate our msi installation process.

WMI namespace — root\CIMV2 has a class – “Win32_Product” which has few methods that work on msi packages. The methods being: —Admin, —Advertise, —Configure, —Reinstall, —Install, —Uninstall,–Upgrade

We will take a look at the Install method to achieve our goal. We can use WMI for automated installation either on a single machine, locally or on a group of machines.


Doing a local computer Installation can be done straight forward with the package location assigned to inParams.

The key here for remote multi machine installation is the “CopyToTempLocation” method that copies the msi from the machine on which the program is running to a local location on the specified remote machine and the new path is then set to inParams.

The outParams[ReturnValue] return the code .i.e the result of execution. There is a comprehensive list of codes returned and the one we should be looking for is  Zero“0” which implies success.

Please visit http://support.microsoft.com/kb/304888 to see the list of codes returned.


Extract details of resources contained within a BizTalk MSI package

Posted on

There are multiple ways to extract details of resources contained within a BizTalk MSI package. One being using msiexec.exe to extract the contents to a location and get the information.

However BizTalk has an excellent dll which can help get all the required resource details that comes in packed in the MSI package.

The saviour here is

Microsoft.BizTalk.ApplicationDeployment.dll that comes with BizTalk installation.

Two dlls are required to be referenced :

1. Microsoft.BizTalk.ApplicationDeployment.dll which has the method that scans for you the MSI for all its resources and then the related properties within each resource. Its placed under the root BizTalk installation folder.

2.Microsoft.BizTalk.Admin.dll from GAC


 Microsoft.BizTalk.ApplicationDeployment.IInstallPackage pack = null;

pack = Microsoft.BizTalk.ApplicationDeployment.DeploymentUnit.ScanPackage(“FullBizTalkMSIPath”);

// Gets all the information about all the resources compliled witin BizTalk MSI package

Microsoft.BizTalk.ApplicationDeployment.IDeploymentResource[] resources = pack.Resources;

foreach (var resource in resources  )

{    // Do something     }

Programmatically Upload and Delete documents from SharePoint library

Posted on Updated on

Documents can be uploaded or deleted from SharePoint library using C# without using SharePoint dlls.
There may arise cases when you want to performs upload and delete documents to and from SharePoint document libraries using C# but you don’t have or want to involve SharePoint dlls to perform this action.
System.Net.WebClient  and System.Net.WebRequest comes in handy in such situation.
These can be used in tandem to achieve the required goal.
// Upload document to SharePoint library
System.Net.WebClient sharePointLibrary = new System.Net.WebClient();
sharePointLibrary.Credentials = new System.Net.NetworkCredential(“SharePointUser“,”SharePointUserPassword“);
byte[] response = sharePointLibrary.UploadFile(Path.Combine(“SharePointURI“, “ResultantFileName“), “PUT“, Path.Combine(“TestLocationofFileToUpload“, “NameOfTheFileToBeUploaded“));
// Delete document from SharePoint library
var request = (System.Net.HttpWebRequest)System.Net.WebRequest.Create(“FullSharePointURIofDocument“);
request.Timeout = System.Threading.Timeout.Infinite;
request.Credentials = new System.Net.NetworkCredential(“SharePointUser“,”SharePointUserPassword“);
request.Method = “DELETE“;
var response = (System.Net.HttpWebResponse)request.GetResponse();