AppDomains, ESB Faults and Context Properties

Last week threw up an interesting problem in the fault message creation module of the BizTalk ESB. The effect of which meant that our fault handler orchestration could not perform callbacks which in turn resulted in orphaned subscriptions. For more detail on the fault handler and subscription model refer to this previous post about the Scatter Gather pattern.

Here's some background information to provide you with some context (pun intended) before explaining the underlying problem and the final solution.

BizTalk Application Domain Unload Thresholds

BizTalk host instance AppDomain creation parameters can be specified in BTNTSvc.exe.config as described here. For the purposes of this post the important things to note in that MSDN article are the assembly unload thresholds named SecondsIdleBeforeShutdown and SecondsEmptyBeforeShutdown. Setting each to -1 will force the AppDomain to never unload assemblies during idle periods (where there are only dehydrated or suspended instances hosted) or periods of inactivity (no instances hosted or interchanges processed at all). However, the defaults for those settings are 20 minutes and 30 minutes, respectively. So, if you have an orchestration dehydrated for 21 minutes and no other interchange activity since the dehydration the AppDomain assemblies will be unloaded. This shutdown behaviour undermines some solution architecture decisions like localised caching, for example.

ESB Fault Message Creation

The ESB Guidance framework provides a set of exception handling helpers that your ESB services use to create fault messages. Messages can be attached to the fault message too. When doing so the ESB will serialise the message and the associated context properties into the fault's property bag. Unfortunately, the BizTalk message object model lets the ESB down at this point as explained below.

Context Properties

Message context properties cannot be iterated over but must be 'fished for' through the API. To fish for context properties you need a .Net type as bait and the most accessible context property type source is the assembly collection already loaded into the current AppDomain...

Note that by iterating over the XLANG segment collection you can actually get access to the context property collection of an XMessage. However, the context properties are in an XmlQName collection. This is insufficient access considering BizTalk only affords an API that takes .Net context property types to reconstitute message context and the XmlQName type doesn't contain fully qualified type names.

The Problem

When the following criteria are met some context properties will not be serialised with the faulted message and a resubmission of the message may not be routeable:

1. Orchestration instance X appends a message to a fault message and it does not have a reference to the property schema assembly of a context property associated with the faulted message; and


2. Orchestration instance X was:
   
   • Dehydrated for a period longer than SecondsIdleBeforeShutdown and there was no further activity after the dehydration before it faulted (in a single server BizTalk group); or
   
   
   • Rehydrated to a load balanced BizTalk server host instance in the group that had recently unloaded it's AppDomain, had recently been restarted and/or has never loaded the required context property schema;

The criteria above may seem rare but can happen if, for example, a timeout occurs waiting for a long-lived transaction to complete.

The Solution

The solution that was decided upon was to simply reload the AppDomain with all the registered property schema types just before the AppDomain was queried by the ESB. This ensures that all possible context property types could be fished for by the fault handler.

See the code snippet below for the LoadPropertySchemaAssembliesIntoAppDomain method call that was added to the fault helper.

namespace Microsoft.Practices.ESB.ExceptionHandling
{
    /// <summary>
    /// Primary helper class for the exception management system
    /// </summary>
    [Serializable]
    public sealed class ExceptionMgmt
    {
        [...]
        private static void CopyMsgToPartProperties(XLANGMessage msg, XLANGPart destMsg)
        {
            [...]
            // Load all the property schema types in case the host instance has since restarted
            // or we have rehydrated to an AppDomain that hasn't any reference to one or more
            // context property types required
            if (_schemaTypes == null || _schemaTypes.Count == 0)
            {
                LoadPropertySchemaAssembliesIntoAppDomain();
            }
            assemblies = AppDomain.CurrentDomain.GetAssemblies();

            // Context Properties: Get the assemblies for property schema types and query the
            // message for those context properties to serialise
            [...]
        }

        /// <summary>
        /// Caches a collection of property schemas obtained using BizTalk Explorer OM.  The idea
        /// behind doing this is to persist all context property schema assemblies in the AppDomain
        /// so they can be queried and matched to message properties of faulted message instances
        /// in <see cref="CopyMsgToPartProperties"/>.
        /// </summary>
        [MethodImpl(MethodImplOptions.Synchronized)]
        private static void LoadPropertySchemaAssembliesIntoAppDomain()
        {
            if (_schemaTypes == null || _schemaTypes.Count == 0)
            {
                try
                {
                    List<Type> schemaTypes = new List<Type>();
                    using (BizTalkQuery operationsQuery = new BizTalkQuery())
                    {
                        bool propertySchemasOnly = true;
                        Collection<BTSchema> propertySchemas = operationsQuery.Schemas(propertySchemasOnly);
                        foreach (BTSchema propertySchema in propertySchemas)
                        {
                            Trace.TraceInformation("{0}: {1}", MethodInfo.GetCurrentMethod(), propertySchema.AssemblyQualifiedName);
                            Type propertySchemaType = Type.GetType(propertySchema.AssemblyQualifiedName, false, true);
                            schemaTypes.Add(propertySchemaType);
                        }
                    }
                    _schemaTypes = new List<Type>(schemaTypes);
                }
                catch (System.Exception ex)
                {
                    EventLogger.Write(MethodInfo.GetCurrentMethod(), ex);
                }
            }
        }
    }
}

Note that the BizTalkOperations service was referenced to query the group's property schema collection. The BizTalkOperations.BizTalkQuery constructor actually instantiates Microsoft.BizTalk.Operations.BizTalkOperations.BizTalkOperations which in turn calls a set of BizTalkMgmt database stored procedures! BizTalkOperations.BizTalkQuery had to be changed to lazy load the BizTalkOperations helper instead, otherwise, the host instance needed BizTalkOperator rights which didn't seem kosher.

If you have come across this issue yourself and have solved it another way I'd be interested in hearing your solution.