BAM - Tracking data using TPE and BAM API

As mentioned in a previous post, I am currently working on a system that moves system generated files to internal and external parties. We had a requirement to provide some visibility to our users for these file movements. We tried to use as many core BizTalk features as possible without creating a lot of custom components. With this said BAM fit in quite well with our requirements. In addition to providing some visibility to our users, we also use BAM to support our auditing requirements.

We also have some requirements that allow our users to re-send a previously sent message. Occasionally some external partners would like a file re-sent. So something that we are also tracking in BAM is the Message ID of the received file. This Message ID does map to the original message in the BizTalkDTADb(live archive or long term archive). For more information about the archives, please see this previous post: http://kentweare.blogspot.com/2007/05/biztalk-retrieving-tracked-messages.html

So one challenge that I did encounter while developing a suitable solution is that there is some data that we want to track via TPE to take advantage of some of the built in features like Business Milestones, Business Milestone Groups, Durations, Measures and Progress Dimensions. The Progress Dimension is a really valuable feature in our design. We have to transfer some large files so our users wanted to see that files were "in progress" while in-flight, completed when finished, and exception when there was a problem sending the file. A Progress Dimension allows you to define these states and then Biztalk will auto-magically track these states.

The problem with using only TPE is that we had some data that did not move through the ports, and therefore cannot be tracked via TPE. In TPE, you typically drag elements/attributes from a schema and map them against a BAM Activity Item. If you try to drag a data element from a schema that did not hit a receive or send port you will get an error indicating that "There are no ports associated with element name.

After digging around the net, I found a very useful article by Richard Seroter http://blogs.msdn.com/richardbpi/archive/2006/03/29/564114.aspx discussing tracking BAM events inside of an orchestration. In his blog he discusses tracking the entire process outside of TPE. In his scenario he is Beginning the Activity inside of a Expression shape. I found that this could work for my requirements, but I also wanted to take advantage of some of the TPE features. If only I could somehow link TPE with Orchestration level tracking.

I did find a way to do this by using "Continuation". Continuation allows you to link multiple segments together. In this context this includes a TPE segment and a Orchestration/BAM API segment.

In my situation, TPE starts the process, inside the orchestration I update the activity with data that did not pass through my receive port and will not pass through my send port and finally TPE will complete the activity.

A question that may come to mind is where am I getting this data from if it does not come through the initial receive port and does not exit the destination send port? I work in the utility industry here in Canada. A specification has been created that defines a naming convention based upon many, many different transaction types. The naming convention includes the following: transaction-type_from-retailer_to-retailer_timestamp.ext

The transaction type and the "to_retailer" make up part of the destination for this particular file. We are able to determine all of the destinations(a file may have more than one destination) by calling the BizTalk Rules Engine. The BRE will evaluate some meta data about the file to be moved and then determine the remaining meta data that is required to send the file out. So there is a lot of information that we need to capture in BAM that is not readily available when we receive the message.

Using the BAM API is a great solution for updating the activity as the message is moving through BizTalk.

So enough banter, here is a snapshot of a portion of my TPE file:

So any element on the left hand side that has a "+" beside it represents BAM data that will be tracked via TPE. Some important items to note here are the StartDateTime, EndDateTime and Exception. These are milestones, that when implemented with a Progress Dimension provide you a near real time status of your business process. In order to configure these milestones, you drag across the appropriate Orchestration shape to match up with the right milestone. So for the StartDateTime, I dragged across the initial Receive shape. You can drag multiple shapes accross to one milestone to represent the various paths that your orchestration can take.

As mentioned above, we want to have the ability to track certain segments using TPE and certain segments using the BAM API. In order to accomplish this we need to create a Continuation Folder. In order accomplish this, right click on the root folder and select "New Continuation". You then need an identifier that you will use to link these segments together. From my initial message I have an element called "ContinuationID" that I have placed in this Continuation Folder. Note this element is not to be confused with TPE's ContinuationID element.

You can now go ahead and apply this tracking profile. When you do you will receive the following warning:

This is ok, because in the Orchestration we will be providing this ContinuationID.

Now lets take a look at what is involved in the Orchestration.

There are a few options when using Orchestration level API calls: DirectEventStream, BufferedEventStream and OrchestrationEventStream. Darren Jefford has provided a summary of these different options which can be viewed at: http://blogs.msdn.com/darrenj/archive/2006/03/17/554049.aspx

I chose to go with the OrchestrationEventStream due to the fact that you do not need to configure a connection string for it and that it utilizes the orchestration persistence mechanism.

So in your BizTalk project, include references to the following assemblies: Microsoft.BizTalk.Bam.EventObservation and Microsoft.BizTalk.Bam.XLANGs. These assemblies may be found in your Microsoft BizTalk Server 2006\Tracking folder.

Since we have already started tracking our business process, via TPE, when we are inside our orchestration, we want to update this activity. In order ensure that our events "line up" between our orchestration and TPE, we need to provide a Continuation Token to ensure that our events from the Orchestration and TPE show up in the same line(record) within our BAM database View/BAM Portal.

This "Continuation Token" is made up of the name of my Continuation Folder(_Continuation) and a unique identifier. This unique identifier is made up of a GUID that is available as part of my inbound message. TPE will track this value and since this value is part of my inbound document I am able to link these two values together. As part of every call that I make I include this continuation token:

continuationToken = "_Continuation" + msg_Internal.ContinuationID;

For each event that I want to track, I call the following "Update Activity" method
Microsoft.BizTalk.Bam.EventObservation.OrchestrationEventStream.UpdateActivity("FileMovements", continuationToken,"FileName",fileName);

Microsoft.BizTalk.Bam.EventObservation.OrchestrationEventStream.UpdateActivity("FileMovements", continuationToken,"ArchiveActivityId",msg_Internal(MessageTracking.ActivityIdentity));

The parameter list for this call is Activity Name, Activity Instance, Activity Item Name, Activity Item Value.

Once you have updated your activity with all of your related data, you can end this portion of the segment by calling the EndActivity method

Microsoft.BizTalk.Bam.EventObservation.OrchestrationEventStream.EndActivity("FileMovements",continuationToken);

So this does not End the entire activity, I rely upon TPE events to complete the entire activity. For instance if a message fails its initial send attempt, it will retry based upon the Send port configuration. While these re-send attempts are occurring, the status of my business process remains in progress until the file is sent correctly.

So as you can see, you do have a few options when tracking BAM data. Obviously there are pros, cons and limitations to both TPE and BAM API calls but you should be able to satisfy your requirements one way or another.

BizTalk - Retrieving Tracked messages through the WMI provider

I am currently involved in a project where we need to provide our users the ability to re-send a file(message). I use the term file as this BizTalk application consumes files from various source systems and sends them to internal/external destination systems primarily as files.

Retrieving tracked messages from HAT is not a complex task, however we want to provide our users with some visibility into what files BizTalk has moved and give them the ability to re-send them if so desired.

There is a class called MSBTS_TrackedMessageInstance2 that allows tracked messages to be saved to disk. This class can be called through a .Net Web App, WinForm app, BizTalk app ..etc.

To keep things simple for this blog, I have included some POC code that is called from a Windows Form app. Be sure to include using System.Management; and include a reference to Microsoft.BizTalk.Operations.

string strDB = "BizTalkMgmtDb";
string strDBServer = "your_server_name_here";
string guidMessageInstanceId = txtGuid.Text;
string strOutputFolder = @"C:\";

try
{
  string strInstanceFullPath =
  "ROOT\\MicrosoftBizTalkServer:MSBTS_TrackedMessageInstance2.MgmtDBServerOverride=\"" 
+ strDBServer + "\",MgmtDbNameOverride=\"" + strDB + 
"\",MessageInstanceID=\"{" + txtGuid.Text + "}\"";


  // Load the MSBTS_TrackedMessageInstance
  ManagementObject objTrackedSvcInst = new ManagementObject(strInstanceFullPath);

  // Invoke "SaveToFile" method to save the message out into the specified folder
  objTrackedSvcInst.InvokeMethod("SaveToFile", new object[] { strOutputFolder });

}

catch (Exception ex)
{
  System.Diagnostics.EventLog.WriteEntry("Get Tracked Message", ex.ToString());

}

When the message is written to disk, both the context xml file and the message are written as two separate files. This action generates the same output that HAT does when you issue the "Save All Tracked Messages" command.

Some of the important things to note in this code is the name of your BizTalk Management Database - BizTalkMgmtDb. This is significant because the name of your Tracking database is found in the BizTalkMgmtDb - adm_Group table.














This is important as your run time BizTalkDTADb cannot grow to infinite size and still be performant. Having this TrackingDBName field does allow you some flexibility as you are then able to specify another BizTalkDTADb which represents a long term archive. You would not modify your primary Management Database to do this. That would break your BizTalk runtime. What you can do is create a copy of your Management Database and then modify this value to represent your long term archive. So you essentially have two management databases and two tracking databases. By creating a copy of the Management database, you can then update this TrackingDBName field to reflect your long term archive tracking database. This allows you to keep your run time BizTalk environment lean and mean while still having long term archived messages.

The long term archive may be the result of an aggregation of your Archive and Purge extracts. There is a tool called the Stitch Utility that will append your Archive and Purge extracts which creates this long term storage archive. For more information regarding this utility check out: http://www.gotdotnet.com/codegallery/releases/viewuploads.aspx?id=67bbd6ea-850e-4d93-be87-df6788976cab

The MessageInstance ID is also a critical parameter in the WMI call as it is able to uniquely identify the message and is based upon a GUID. The MessageInstance ID can be retrieved via a couple methods. One way is through HAT. When you view the Message flow for a particular message, the MessageInstance Id is one of the context properties that is tracked.
















Another way of getting this value is through the context properties of the incoming message. Suppose you have an inbound message called msg_In. You could retrieve the id for this message by getting the value of msg_In(BTS.MessageID).

As part of our project, we track this MessageInstanceId in BAM. We have a custom ASP.Net web page that queries the associated BAM view in order to retrieve the MessageInstance ID. One nice thing about tracking the information in BAM is that we can also track other meta data about the file movement such as the File name, Date/Time, Source/Destination locations and several other fields. This allows our users to query upon several different parameter combinations in order to find the particular file(message) that they are interested in.

I plan on blogging more about the BAM portion of this solution in an upcoming post.