Large Message Persistence In BAM

BAM is all about interchange transparency. Whether you're looking for operational instrumentation, business process monitoring, exception logging, messaging trend analysis, data aggregation or as a nonrepudiation database BAM, in my opinion, should be your weapon of choice.

The EventStream API is the most powerful of all the interfaces into BAM. The buffered, direct, orchestration and pipeline (messaging) variants offer support for performant, race condition and transactional applications, respectively.

Accomodating large message persistence to a BAM data store is afforded but with restriction.

To explain, EventStream.AddReference has an overload which facilitates large message persistence. However, it accepts a System.String parameter for 'long reference data'. BAM documentation reads that this method has a reference data limit of 512KB but the underlying SQL data type is national text (ntext). One would assume that the 512KB limitation is simply advisory because larger messages can be written through this interface.

It is appropriate now to quantify 'large messages'. A governed SOA environment will limit or eliminate interchange of messages > 50MB through BizTalk by either choosing another technology or provisioning BizTalk endpoint contracts to optimise efficiency. 95% of large messages can be transacted through EventStream.AddReference but the other 5% of large messages can often cause system resource issues. This thread focuses on that 5%.

A common response by solution teams to high system resource consumption by their product is to add more processors or RAM to the BizTalk server(s). This is justified in some cases, however, frequently this is not appreciated by Operations or the business and may actually require more than just simply adding hardware, for example, BizTalk licencing extensions or an OS upgrade to accomodate more than 4GB RAM.

.Net 2.0 to the rescue, the System.IO.Compression namespace in particular. The GZipStream class to be more specific.

BizTalk is inherently stream-based. Seeing that AddReference only accepts strings it is important to also remember the 'acquire late, release early' programming tenet to reduce the foot print of the message persistence process. XLangPart.RetrieveAs(typeof(Stream)) is a powerful method call and leverging this for applying message compression is the crux of being able to successfully write large messages through the BAM API.

When retrieving the message from BAM the DocumentUrl reference type can be used to include a compression state indicator so your custom message viewer can easily identify how to render the message.

BAMManagementService.GetReferences can be called to retrieve the stored message from the BAMPrimaryImport database. Unfortunately, to leverage this webmethod you will need to encode the compressed stream before writing it into BAM otherwise the SOAP request will fail with a hexadecimal character fault when the response is constructed. Base64 encoding and decoding is fast but expensive in that it adds 33% expansion to the compressed byte stream. A message size compression threshold might be useful here to save unnecessary expansion and processing overhead for smaller messages. To reduce strain on the BAM portal when transmitting large messages back to the client one can use a file download threshold to decide whether to stream the decompressed message.

No code samples this time.

Enjoy