BizTalk 2006 - Missing Tracking Data (HAT and BAM)

I was reluctant to publish post, but have decided to because at the end of the day I hope this post saves some people time and frustration. However, I won't be able to sleep at night without posting the following disclaimer:

*********************************Disclaimer**********************************
This post describes invasive techniques that will remove data from your BizTalk Databases. Some of these procedures may not be supported, by Microsoft, in production environments. Also, when in doubt please contact Microsoft Premiere support. Proceed at your own risk.
******************************************************************************
In our Test environment we were experiencing missing BAM data. The processes that BAM was tracking were completing successfully. We tried to reproduce the problem in other environments by deploying the BizTalk project and BAM artifacts to other servers. We were unable to re-produce the problems in these environments. That eliminated any coding issues as this project has been live for some time and a 2nd iteration was just being implemented.

Knowing that the issue was not code related forced us to focus on the current infrastructure. Since I was not on this project, I started running some tests hoping to re-produce the problem. This did not become too much of an issue as submitting as little as 8 messages to BizTalk would create some anomalies.

Anomaly #1 - Missing HAT Data

While trying to reproduce the problem, I noticed that we were missing some Tracking data related to HAT.

We would see instances that had started and that "apparently" had not completed. When stepping through the orchestration debugger there were some common shapes where the tracking would just stop.

However, within this process we have several log points where we will insert a log event into the event viewer. We also are communicating with a few different systems and were able to determine that these downstream systems were receiving messages from BizTalk.

Anomaly #2 - Missing BAM Data
Within the BAM Portal we would find columns that did not contain any information. We are using milestones, so the process was completing yet, there was data that was missing. Within this process we are using both the BAM API and TPE. TPE only tracks data that hits ports where as we had some information that never hits a port which forced us to use the BAM API.



What did not add up was that for an instance that was completed, and had missing data, the Expression shape that included the BAM API Call was being executed.

At this point I thought that the two anomalies could be related since we are taking advantage of the OrchestrationEventStream within the API. Darren Jefford has a great explanation of the OrchestrationEventStream here so I am not going to try and come up with something better here.

After some further investigation I was not getting too far so I figured it was time to reach out for some technical support. What I was told was that: "the TDDS sequence numbers in TDDS_SteamStatus tables and TrackingData tables have gotten out of sync." This will result in the TDDS subservice discarding Tracking data due to the synchronization issue. This definitely explains the missing data in HAT.

The previous description is more directly related to the missing HAT data. Since the BAM data is essentially Tracking data and following the same process as HAT data, due to the OrchestrationEventStream, it was plausible that both events were completely related.

Here are the steps that we performed to solve our issue:
(Disclaimer in effect - Please execute caution before running these steps. There is no way to be 100% sure that your issue was the same as mine. If you are unsure, contact Microsoft.)

• Stop all BizTalk Host Instances
• Stop SQL Agent (These two steps will ensure that no one is connecting to the database(s) )
• Ensure that you have no Active, Dehydrated, or ready to run instances in BizTalk Admin
• Verify that the TrackingData table has a row count of zero (You will need to drain any remaining messages by starting your tracking host back up, turn it off when row count gets to zero)
• Backup the Management, MsgBox, DTADb(Tracking), BAMPrimaryImport databases
• Truncate the following ables: BizTalkDTADB.TDDS_StreamStatus, BAMPrimaryImport.TDDS_StreamStatus

After performing these actions, we were set to run some tests to try and reproduce the issue. After running several scenarios and putting greater than production loads on the environment we still could not re-produce the issue.

What is still concerning about this issue is how it originally happened. There were no obvious changes to our environment that prompted this issue. We were also lucky that the event occurred in the the Test environment as rebuilding the environment was on the table at one point.

BizTalk 2006 - Message Viewer HAT Plugin

A tool that I have been using for a few months now, that I have not had a chance to blog about, is the Message Viewer HAT Plugin. The tool is available for free download on codeplex and was written/posted by someone with the handle of faizelk.

What does the tool do?
The tool allows you to view messages from within the Health and Activity Tracking (HAT) tool without having to save each message to disk.

Why should I use this tool?
If you have every played the "blame" game then you know that searching for messages inside of HAT can be a time consuming process. Especially if you don't have a promoted property to limit your search. You usually end up saving messages to disk to inspect them. This goes on for a bit until you actually find the message that you are looking for.

This tool will save you a lot of time as you don't have to leave the HAT tool to actually view message bodies. This of course assumes you have enabled message body tracking.

For those of you who are unfamiliar with the "blame" game that is when you are on an integration project with multiple parties and the finger pointing begins. HAT, with message body tracking enabled, becomes the tool of choice because you can provide "the evidence" of what BizTalk did. The conversation usually goes something like this: "BizTalk did what it was suppose to do, this is the message that I received and this is what I sent". The stakes and intensity of the game usually increase with the number of participants and systems involved. :-)

Install process
It is a simple process:

• Download the zip file from here


• Extract the Bizspoke.BizTalk.HATPlugIn.dll to the following BizTalk directory: X:\Program Files\Microsoft BizTalk Server 2006\Tracking\Plugins where 'X:' is the drive in which you have installed BizTalk on.

Running the Tool
The first thing you will need to do is launch HAT. Once HAT is available, click on "Queries" - "Most recent 100 service instances". Modify your query if desired.

Right mouse click on your desired message. You will now see a new item added to the menu. Click on the item called "(Bizspoke) View tracked messages...".

A .Net form will launch that will contain the "goods". From within this form you will find:

• The Message Body displayed on the right hand side


• The Message's context properties in the lower left


• The ability to Save the Message(s). This function works the same way that it does from within HAT; it will save all of the messages and context files


• The ability to view the XML file in Internet Explorer


• If you have selected a bunch of messages that you want to view, you can use the "Previous" and "Next" buttons to navigate through them.

Consider the scenario where you are tracking messages before and after port processing. These scenarios would include dealing with Flat Files where you have built a Flat File schema and are calling it in a Pipeline or when using the SAP Adapter to receive/send IDocs. In these scenarios, you have the ability to choose which message(s) that you want to track from within the Port - Tracking configuration in BizTalk Admin. If you choose to track both messages, the Message Viewer Plug-in will list both messages in the left frame and you can then choose which file you want displayed in the message body frame.

In my example, I have tracked a SAP IDoc(which is essentially a flat file) so it will get displayed in the "Text" tab. This message represents one that has been tracked before port processing (I received the message).

The second message body is the XML version of the IDoc; the one that BizTalk will use internally. By selecting this message and the "XML" tab, I can see it in a XML friendly format.

BizTalk Versions?
I have been using this tool with BizTalk Server 2006 R1. We have it deployed in all of our environments including Production - it is that valuable. If you have been keeping up with this blog, you may have noticed that I have a post regarding HAT's status in BizTalk 2009 Beta. For kicks, I copied the dll into the equivalent folder in my BizTalk 2009 installation folder and it didn't have any affect(it does not show up in any BizTalk Admin Console menus). So I am not sure how this tool will fit in moving forward, but if you are running BizTalk Server 2006, I recommend installing this tool.