BizTalk ESB Scatter Gather Framework

I have recently been working on a BizTalk ESB project which uses the scatter gather pattern heavily. One of the key requirements was to execute itineraries in parallel. This thread is an overview of the framework written to fit the requirement.

The scatter gather model chosen was taken from the ESB sample but extended to allow advanced correlation, abstraction of the broker and dispatcher components and fault management.

Walking the diagram above (left to right) the following components are invoked:

Abstract Broker

This component is abstract in that more than one implementation can fill the broker role. The broker role being the primary distribution and subsequent aggregation of messages. Examples of brokers that are useful are:

1. Debatching - Use a pipeline to debatch individual messages within an envelope and distribute to a dispatcher


2. Multi-endpoint - Loop through a resolver collection distributing one message to multiple different dispatchers


3. Sequential - A debatching broker that waits for a gather response before scattering the next message

Each broker adds it's own features to or re-organisation of common steps/shapes to implement the behaviour required.

Dispatch Correlator

The correlator is the glue between the broker and dispatcher. It does the following:

1. Rewrap the dispatch correlation request payload in a dispatch request


2. Write the following into the message context of the dispatch request: Thread ID, Dispatch Type and an IsInScatterGather boolean


3. Relay the message to the message box after initialising a correlation set on the thread ID and dispatch type


4. Relay the response received from the dispatcher, by following the correlation set, back to the broker

The message context values written in step 2 are used by the Fault Manager, explained later.

Abstract Dispatcher

Dispatchers are tightly coupled to an endpoint but loosely coupled to the broker. An implementation must always respond to the dispatch correlator to prevent orphaned processes. Examples of dispatchers that are useful:

1. Itinerary - Uses the dispatch address to locate the itinerary to stamp to the message. An Itinerary Callback orchestration is paired with this dispatcher to callback into the dispatch correlator and must be configured as the final step in the itinerary that is spawned


2. Loopback - Simply returns the message received. Used to allow the broker to relay the request in the aggregated response

Dispatchers must ensure that exceptions are handled by either responding to the dispatch correlator directly or indirectly by publishing a fault message with a copy of the context properties from the inbound message.

Fault Manager

The fault manager subscribes to routed failed messages (FMR) and checks the message context for the IsInScatterGather boolean set by the dispatch correlator. If the context property is available and it's true then a dispatch correlation response is constructed to indicate a fault has occurred before being written to the message box.

Notes

There are a few important points to note about the model above:

1. No delays are implemented in the framework to 'protect' it from orphaning threads


2. Exception handling through the stack must be bullet-proof, period


3. Contrary to typical ESB service development, strongly typed messaging between the components in the framework is key

The framework is extremely flexible. When using the itinerary dispatcher the ESB is transformed from a serial to a parallel bus. Also being explored at the moment is using the framework as a platform for the resequencer pattern.