BizTalk 2006 - Missing Tracking Data (HAT and BAM)

I was reluctant to publish post, but have decided to because at the end of the day I hope this post saves some people time and frustration. However, I won't be able to sleep at night without posting the following disclaimer:

*********************************Disclaimer**********************************
This post describes invasive techniques that will remove data from your BizTalk Databases. Some of these procedures may not be supported, by Microsoft, in production environments. Also, when in doubt please contact Microsoft Premiere support. Proceed at your own risk.
******************************************************************************
In our Test environment we were experiencing missing BAM data. The processes that BAM was tracking were completing successfully. We tried to reproduce the problem in other environments by deploying the BizTalk project and BAM artifacts to other servers. We were unable to re-produce the problems in these environments. That eliminated any coding issues as this project has been live for some time and a 2nd iteration was just being implemented.

Knowing that the issue was not code related forced us to focus on the current infrastructure. Since I was not on this project, I started running some tests hoping to re-produce the problem. This did not become too much of an issue as submitting as little as 8 messages to BizTalk would create some anomalies.

Anomaly #1 - Missing HAT Data

While trying to reproduce the problem, I noticed that we were missing some Tracking data related to HAT.

We would see instances that had started and that "apparently" had not completed. When stepping through the orchestration debugger there were some common shapes where the tracking would just stop.

However, within this process we have several log points where we will insert a log event into the event viewer. We also are communicating with a few different systems and were able to determine that these downstream systems were receiving messages from BizTalk.

Anomaly #2 - Missing BAM Data
Within the BAM Portal we would find columns that did not contain any information. We are using milestones, so the process was completing yet, there was data that was missing. Within this process we are using both the BAM API and TPE. TPE only tracks data that hits ports where as we had some information that never hits a port which forced us to use the BAM API.



What did not add up was that for an instance that was completed, and had missing data, the Expression shape that included the BAM API Call was being executed.

At this point I thought that the two anomalies could be related since we are taking advantage of the OrchestrationEventStream within the API. Darren Jefford has a great explanation of the OrchestrationEventStream here so I am not going to try and come up with something better here.

After some further investigation I was not getting too far so I figured it was time to reach out for some technical support. What I was told was that: "the TDDS sequence numbers in TDDS_SteamStatus tables and TrackingData tables have gotten out of sync." This will result in the TDDS subservice discarding Tracking data due to the synchronization issue. This definitely explains the missing data in HAT.

The previous description is more directly related to the missing HAT data. Since the BAM data is essentially Tracking data and following the same process as HAT data, due to the OrchestrationEventStream, it was plausible that both events were completely related.

Here are the steps that we performed to solve our issue:
(Disclaimer in effect - Please execute caution before running these steps. There is no way to be 100% sure that your issue was the same as mine. If you are unsure, contact Microsoft.)

• Stop all BizTalk Host Instances
• Stop SQL Agent (These two steps will ensure that no one is connecting to the database(s) )
• Ensure that you have no Active, Dehydrated, or ready to run instances in BizTalk Admin
• Verify that the TrackingData table has a row count of zero (You will need to drain any remaining messages by starting your tracking host back up, turn it off when row count gets to zero)
• Backup the Management, MsgBox, DTADb(Tracking), BAMPrimaryImport databases
• Truncate the following ables: BizTalkDTADB.TDDS_StreamStatus, BAMPrimaryImport.TDDS_StreamStatus

After performing these actions, we were set to run some tests to try and reproduce the issue. After running several scenarios and putting greater than production loads on the environment we still could not re-produce the issue.

What is still concerning about this issue is how it originally happened. There were no obvious changes to our environment that prompted this issue. We were also lucky that the event occurred in the the Test environment as rebuilding the environment was on the table at one point.

BizTalk 2006 - Message Viewer HAT Plugin

A tool that I have been using for a few months now, that I have not had a chance to blog about, is the Message Viewer HAT Plugin. The tool is available for free download on codeplex and was written/posted by someone with the handle of faizelk.

What does the tool do?
The tool allows you to view messages from within the Health and Activity Tracking (HAT) tool without having to save each message to disk.

Why should I use this tool?
If you have every played the "blame" game then you know that searching for messages inside of HAT can be a time consuming process. Especially if you don't have a promoted property to limit your search. You usually end up saving messages to disk to inspect them. This goes on for a bit until you actually find the message that you are looking for.

This tool will save you a lot of time as you don't have to leave the HAT tool to actually view message bodies. This of course assumes you have enabled message body tracking.

For those of you who are unfamiliar with the "blame" game that is when you are on an integration project with multiple parties and the finger pointing begins. HAT, with message body tracking enabled, becomes the tool of choice because you can provide "the evidence" of what BizTalk did. The conversation usually goes something like this: "BizTalk did what it was suppose to do, this is the message that I received and this is what I sent". The stakes and intensity of the game usually increase with the number of participants and systems involved. :-)

Install process
It is a simple process:

• Download the zip file from here


• Extract the Bizspoke.BizTalk.HATPlugIn.dll to the following BizTalk directory: X:\Program Files\Microsoft BizTalk Server 2006\Tracking\Plugins where 'X:' is the drive in which you have installed BizTalk on.

Running the Tool
The first thing you will need to do is launch HAT. Once HAT is available, click on "Queries" - "Most recent 100 service instances". Modify your query if desired.

Right mouse click on your desired message. You will now see a new item added to the menu. Click on the item called "(Bizspoke) View tracked messages...".

A .Net form will launch that will contain the "goods". From within this form you will find:

• The Message Body displayed on the right hand side


• The Message's context properties in the lower left


• The ability to Save the Message(s). This function works the same way that it does from within HAT; it will save all of the messages and context files


• The ability to view the XML file in Internet Explorer


• If you have selected a bunch of messages that you want to view, you can use the "Previous" and "Next" buttons to navigate through them.

Consider the scenario where you are tracking messages before and after port processing. These scenarios would include dealing with Flat Files where you have built a Flat File schema and are calling it in a Pipeline or when using the SAP Adapter to receive/send IDocs. In these scenarios, you have the ability to choose which message(s) that you want to track from within the Port - Tracking configuration in BizTalk Admin. If you choose to track both messages, the Message Viewer Plug-in will list both messages in the left frame and you can then choose which file you want displayed in the message body frame.

In my example, I have tracked a SAP IDoc(which is essentially a flat file) so it will get displayed in the "Text" tab. This message represents one that has been tracked before port processing (I received the message).

The second message body is the XML version of the IDoc; the one that BizTalk will use internally. By selecting this message and the "XML" tab, I can see it in a XML friendly format.

BizTalk Versions?
I have been using this tool with BizTalk Server 2006 R1. We have it deployed in all of our environments including Production - it is that valuable. If you have been keeping up with this blog, you may have noticed that I have a post regarding HAT's status in BizTalk 2009 Beta. For kicks, I copied the dll into the equivalent folder in my BizTalk 2009 installation folder and it didn't have any affect(it does not show up in any BizTalk Admin Console menus). So I am not sure how this tool will fit in moving forward, but if you are running BizTalk Server 2006, I recommend installing this tool.

BizTalk 2009 Beta - Where's my HAT?

The Health and Activity Tracking tool, otherwise known as HAT is no longer available in BizTalk 2009. Well at least not in its current Beta form. The HAT tool allowed BizTalk developers/admins the ability to view BizTalk's "processing logs". You could see when a message was received, when it was processed by an orchestration (if applicable) and when the message was sent to a downstream system.

You also had the ability to customize the default queries and save them if you wished. Provided you had message body tracking enabled, you could also save the message bodies and message context from HAT.

At least for now, you willl no longer see the HAT shortcut in your Windows Start Menu. I assume that the change is permanent but since this is beta you never know.



The HAT executable (BTSHatApp.exe) still exists in your BizTalk install folder (x:\Program Files\Microsoft BizTalk Server 2009), however you cannot launch it as it will provide an error.

Harold Hoffman was able to get the tool to launch from the command line by providing the right parameters but the tool itself doesn't really allow you to do much with it.

So with the disppearance of HAT, have we been hung out to dry? The simple answer is No! The HAT functionality that we have all grown to love and hate can now be found in the BizTalk Admin console. This functionality can be found from the Group Hub Overview page. To create a new Query click on the "New Query" tab.

Note: For the following images right mouse - > Open Link in New Window for a larger image


You can narrow your search by specifying the types of BizTalk Service instances that you are interested in. For instance if you were only interested in Suspended Services instance you could select this value from the "Value" drop - down.


If you are interested in finding a message that was previously received or sent then selecting "Tracked Message Events" from the drop - down would be a good choice. From here you can also save the message body, if message body tracking was enabled, by right mouse clicking on a result and selecting "Save to File".

If you want to further constrain your results, you can add criteria to your search. In my scenario I have added "Schema Name" to the "Field Name" column. When you click on the "Value" drop - down you will see a list of deployed schemas to select.

Dealing with adding dates into your HAT query was always a pain; you had to account for you local timezone when constructing your query. This either meant using the DateAdd function or manually doing the timezone calculation in your head. Nice to see that they have made this process smoother.

A calendar control is included which makes date restriction much easier than in the HAT custom query window.

As you can see in the image below, that I didn't have to make any accommodations for GMT and local timezones.


If you were looking for more than just messages (say an orchestration) you could do a search for "Tracked Service Instances". You can further restrict your results by including additional criteria from the "Field Name" drop - down. For instance if you could restrict your results to a particular BizTalk Host.

Within the BizTalk Admin Console, the product team has included some convenience features like the calendar control and pre-populating drop - down controls. I did notice an inconsistency between the Host Name drop downs when performing searches against "All In-Process Service Instances" and "Tracked Service Instances".

As you can see BizTalkServerApplication and BizTalkServerIsolatedHost have been pre-populated when performing searches for "All In-Process Service Instances".




In the case of "Tracked Service Instances" the Host Names are not pre-populated. When I did insert the name of my Host and clicked the "Run Query" button it did return my desired result. I am not sure if this is a bug or design decision but remember...its Beta. I have submitted this to the Microsoft Connect site and we will see what happens with it.



Conclusion
Moving the HAT functionality into the BizTalk Admin console was definitely the right decision. Having to use 2 different tools from BizTalk 2006 onward was a bit of a pain. The BizTalk Admin console was a HUGE step forward from BizTalk 2004 so to have all of this functionality in one tool is great. After using HAT for almost 5 years, it will take a little getting use to but I am looking forward to using the BizTalk Admin console for all of my historical tracking needs.

What version of BizTalk do I have?

I have seen this question asked several times in the forums and newsgroups so I figured it would be worth posting. I guess it relates more to determining whether you have BizTalk 2006 R1 or BizTalk 2006 R2 installed as BizTalk 2009 has some noticeable branding differences that R1 and R2 do not have.

The content itself comes from the BizTalk 2009 MSDN documentation, but I am posting for quick reference and in case the MSDN link changes(it is beta documentation).


To determine your version, do the following:

1. From RegEdit, navigate to the HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\BizTalk Server\3.0 key.


2. Check the ProductVersion key and note the version number.


3. Map the version number from your key to the table below:

3.0.4902.0 BizTalk Server 2004
3.0.6070.0 BizTalk Server 2004 SP1
3.0.7405.0 BizTalk Server 2004 SP2
3.5.1602.0 BizTalk Server 2006
3.6.1404.0 BizTalk Server 2006 R2

If the value of the ProductVersion key is greater than 3.6.1404.0, the version of BizTalk Server is BizTalk Server 2009.

On my machine I am running BizTalk 2009 Beta and my version is 3.8.104.5 but do remember that this is a Beta release and this version number is bound to change.

Microsoft MVP Awarded: Part 2

I was recently informed that I have received MVP status from Microsoft for the 2nd consecutive year. The MVP experience is a great one; it is a lot of hard work, but provides excellent opportunities to exchange information with Microsoft employees and other MVPs.

The highlight of my first year was definitely attending the MVP Summit and interacting with both the Product Group and other MVPs from all of the world. I met some great people there and continue to stay in regular contact with many of them.

If you would like to see my MVP profile, you may see it here.

BizTalk 2009 Beta - Code Migration Process

After going through the process of migrating a BizTalk 2006 R1 solution to BizTalk Server 2009 (beta) I am happy to report that the process is a painless endeavor. The process itself reminds me a lot of the BizTalk 2004 - 2006 migration process.

Scenario
My dev box has a clean install of BizTalk Server 2009 and did not previously have Visual Studio.Net 2005 on it. Therefore, I needed to go through the Code Conversion process. I used a BizTalk 2006 project from a previous blog post. The project was not all that complex but it did include the typical artifacts that you would find in a BizTalk solution. This includes:

• The BizTalk Solution file
• Orchestration Project
• MapsAndSchema Project
• Pipeline Project
• .Net Helper Project
• Binding Files (more on this later)

Process
As you can see the DeliveryNotification sln file is a Visual Studio '8' (or VS.net 2005) file.

Double clicking on this file will launch Visual Studio 2008 and the Code Conversion Wizard

After clicking on "Next" you will be prompted with a dialog box that lists all of the related projects that will get migrated.

Success! Enable the check box to see the conversion report.

The results of the conversion process are documented below and the project listings are on the right hand side. Note that BizTalk projects currently have a "C#" icon. I do expect this to change as this is Beta. The underlying reason for this is that BizTalk projects are now considered to be 'flavours' of C# projects as discussed in this post.

So what was converted?
Here is a screenshot of what my BizTalk artifacts look like after the conversion process. By observing the Date modified timestamps, you get a sense of what was changed.

After taking a closer look, you will find that the actual BizTalk artifacts(schemas, orchestrations, maps etc) are not actually getting modified. Rather, it is the Visual Studio artifacts(solution file, project file, user settings) that are being modified.

I look at this as good news/bad news. Good in the sense that you will have a lot confidence that nothing will get broken when converting code. Bad in the sense that this means little has changed(from a development perspective) between BizTalk Server 2006 Rx and BizTalk Server 2009

Post Conversion
So the conversion process worked - now what? When I tried to build and deploy the solution I had no issues:

When I launched the BizTalk Admin Console, I was pleased to see that my Application Container and related artifacts were created during my deployment from VS.net 2008. Since this was my initial deploy on a clean dev box, I didn't have any bindings for the orchestration, receive locations or send ports.

I decided to import an existing binding file from my BizTalk Server 2006 solution.

The binding file import was successful!

I did discover an issue when I tried to start up the application from the tree view on the left hand side:

I could not Start, Stop or Configure the application. I was able to manually start the send ports, the orchestration and enable the receive locations. Or I could click on "Applications" folder and start the application from the "main frame". Both of these actions did start the application and turned the application's icon green.

I checked the Microsoft Connect site and saw that Ben Cline had already reported this as a bug. I added some of my own comments so Microsoft is aware of it and I would expect that it will get resolved for the next release.

Once the application was online, I processed some files to ensure that the application was functioning correctly and that it was.

Conclusion
While my project was not all that complex, I was happy with the migration experience and feel pretty confident in moving my projects from BizTalk 2006 to BizTalk 2009.

BizTalk 2009 Beta - First Look

On December 8th, 2008 I created a blog post indicating that the BizTalk 2009 Beta had gone public. Since then I have had a chance download the bits and play around with this latest release from Redmond.

I consider this release to be an evolutionary release, not an revolutionary release. One of the main purposes of this release is to align with Microsoft's latest technology platform offerings. This includes: Windows Server 2008, SQL Server 2008 and Visual Studio 2008. In addition to those additions, you will also see tight integration with Team Foundation Server, ESB Guidance 2.0, greater Unit Testing and debugging abilities and additional UDDI support.

I have taken some screen shots of the installation experience and will provide some relevant comments inline. The install was done on a Windows Vista dev PC, using SQL Server 2008 Developer edition and Visual Studio 2008 Professional Edition.

Also note that this is Beta code so there are bound to be some changes to the final cut of the code.

The Prerequisite component installation remains a pain-less and efficient endeavor. Who remembers the pain that gathering all of the BizTalk Server 2004 prerequisites inflicted?

Something you may notice missing is Human Workflow Services (HWS). This feature was previously deprecated in BizTalk Server 2006 so it does not come as too much of a surprise that it is M.I.A.

For continuous integration enthusiasts, the highlighted "Project Build Component" should be of interest. It allows you to build BizTalk projects on a server that does not have an installation of Visual Studio on it. For instance, within my organization we use CruiseControl.net to poll our VSS repository looking for any changes to BizTalk projects. This occurs on a dedicated Build Server where we have an installation of VS.Net. Using this feature would allow us to reduce a VS.Net license. BizTalk is a first class citizen in MSBuild. So everything that you can do in VS.Net (with respect to build and deploy) you can do in MSBuild.

Below is the standard configuration screen that we are use to seeing with BizTalk Server 2006 Rx. Not too much has changed with the exception of MSMQT. This was another feature that was deprecated in BizTalk 2006. Microsoft has been pretty adamant about getting off of MSMQT as there were plans for its dismissal

Success...and on the first attempt no less!

Since BizTalk was installed on a machine with VS.Net already on it, you can see that the BizTalk templates have now been installed which allows developers to create and modify BizTalk Projects.

So as I have previously stated, this is an evolutionary release so you will not find many actual BizTalk changes in Visual Studio. This includes:

• No new BizTalk Shapes
• No new Functoids
• The Expression Editor still is not re-sizable - This debate never gets old :-)

Something that you will notice different is that the "Development and Deployment" VS configurations are gone! You will now only have Debug and Release which aligns better with what .Net programmers are use to. It also simplifies some of the automated Build and Deploy configurations, when dealing with .Net Helper assemblies.

I had the opportunity to participate in a Microsoft BizTalk 2009 Brown Bag session and have made some ad-hoc notes which I am happy to share with you (Non-NDA):

• BizTalk Project System – New Project System is implemented as a Flavor of C# Project System. This should allow for smoother transitions for future versions of Visual Studio. The BizTalk project is now a first class citizen in Visual Studio.
• BizTalk Project can contain any item (not just BizTalk artifacts). This means that you can have a C# class in your BizTalk project and do not have to create a separate .Net project that BizTalk would call. Note there are a few minor issues with adding a new class in the current build of this Beta but it will be fixed.
• Debug and Release as default configurations as opposed to Development and Deployment
• Add Web Reference is not directly available on Project anymore. Need to access it through “Add Service Reference” dialog box.
• Migrated all BizTalk project and item templates to the new style (.vstemplate)
• Old style templates are still supported.
• Full Visual Studio integration for BizTalk project migrations. This integration supports both BizTalk 2006 (R1 and R2) to 2009 migrations. ( I plan on creating a separate blog post that deals specifically with migration).
• During migration, only .sln, btproj and btproj.user files are backed up as there are no fundamental changes to remaining artifacts.
• Support for conversion of projects checked into Source Code Control.
• Implemented entire BizTalk build system using MSBUILD.
• Support for incremental builds (only re-build what you need to rebuild - this will reduce some of the build times for minor changes to your project).
• Refactored all compilers to eliminate any VS Dependencies – can compile BizTalk projects without VS – Project Build Component in BizTalk install (for build server).
• Map, Schema(validate against sample instances) and Pipeline(programmatically) unit tests can now be executed from within Visual Studio.
• Currently you cannot debug a map that has an extension object – such as calling an external assembly – will be addressed in a future release. You can unit test it though.
• Currently(2009 beta) multi source maps cannot be united tested
• DEBUG and TRACE constants will work for c# classes in BizTalk projects.