BizTalk Adapter 3.0 vs SAP Enterprise Services

For those of you who are interested in integration with SAP, there is a document worth reading called Positioning SAP Enterprise Services vs Microsoft BizTalk Adapter 3.0 for mySAP Business Suite. The document itself describes the different scenarios in which you can use the BizTalk Adapter Pack 3.0 when integrating with SAP. These scenarios include:

• .Net Based Applications
• SQL Server 2005 Integration Services
• SQL Server 2005 Reporting Services
• Sharepoint
• and of course BizTalk.

In case you are not aware, the BizTalk Adapter packs are based upon WCF and therefore can be run outside of BizTalk even though they are branded "BizTalk". The Adapter pack itself is included in the license of BizTalk, but you cannot run the Adapter pack on a server without BizTalk unless you have a separate license for it or you do not have BizTalk installed. See the Adapter pack pricing page for more licensing info.

In my last post, I discussed how the SOAP and SQL Adapters have been deprecated for BizTalk 2009. Well, you can add another adapter to the list and that is the Microsoft BizTalk Adapter v2.0 for mySAP Business Suite. This is the adapter that originated in BizTalk 2006 R1 on the Microsoft BizTalk Adapters for Enterprise Applications cd.

The primary objective of this document is to discuss the scenarios where you would want to use the BizTalk adapter versus when you would want to use the Web Service capability of Enterprise Services. SAP ECC 6.0 natively provides Web Service capabilities. So this document is probably to address those opinions that if SAP has Web Services then why do I want to use BizTalk,or the Adapter pack, to exchange data between SAP and another application?

Here are a couple of ideas that I can think of:

• Avoid point to point integration scenarios
• Take advantage of BizTalk's transformation(mapping) capabilities
• Durable messaging (retries, persistence)
• Monitoring(MOM,SCOM)/Tracking(HAT,BAM)
• Managing a Business Process between multiple systems.
• Adapter provides reliable, performant and secure data transmissions.

BizTalk 2009 Beta Documentation - now available

The BizTalk 2009 Beta Documentation is available here. (Thanks Mick Breeze / Rahul Garg ).

While 'flipping' through the documentation I discovered an interesting section: Deprecated Features, Tools, and APIs. Within this section, the following features have been marked as deprecated:

• BizTalk Explorer
• SOAP Adapter
• SQL Adapter

For those of you who may be new to BizTalk, BizTalk Explorer is a view that is available inside Visual Studio that allows you to connect to your BizTalk Management database. Personally, I have not used this view since BizTalk 2004, but there may be some out there that started using it in 2004 and never let it go.

SOAP and SQL Server connectivity will still be supported in the future by BizTalk, but by using a different adapter. Basic Profile Web Services are a supported scenario in the base WCF adapter so any integration with basic Web Services should be done using the WCF-BasicHttp adapter instead of the SOAP adapter.

The BizTalk Server Adapter pack 2.0 includes a sqlBinding that allows you to communicate with SQL Servers using the WCF adapter. Currently, the supported versions of SQL that you can communicate with include SQL 2000, 2005 and 2008.

The guidance around deprecated features is that you should not use them when developing new applications. This also gives you a transition period to move off of these features before the next release of BizTalk (post 2009).

A bi-product of these features being deprecated are the tools that compliment these features. The tools being deprecated include:

• BizTalk Web Services Publishing Wizard
• Add Web Reference

Both of these tools have their equivalent WCF versions so you will use these exclusively in the future.

Within the BizTalk 2009 release, we have seen a few features removed including MSMQT and HAT that had been previously deprecated. So we know that Microsoft means business when they provide these early warnings. So my advice to you would be to start using the WCF adapters where possible so that you are not re-factoring your apps at the last minute in the future.

BizTalk 2006 - Missing Tracking Data (HAT and BAM)

I was reluctant to publish post, but have decided to because at the end of the day I hope this post saves some people time and frustration. However, I won't be able to sleep at night without posting the following disclaimer:

*********************************Disclaimer**********************************
This post describes invasive techniques that will remove data from your BizTalk Databases. Some of these procedures may not be supported, by Microsoft, in production environments. Also, when in doubt please contact Microsoft Premiere support. Proceed at your own risk.
******************************************************************************
In our Test environment we were experiencing missing BAM data. The processes that BAM was tracking were completing successfully. We tried to reproduce the problem in other environments by deploying the BizTalk project and BAM artifacts to other servers. We were unable to re-produce the problems in these environments. That eliminated any coding issues as this project has been live for some time and a 2nd iteration was just being implemented.

Knowing that the issue was not code related forced us to focus on the current infrastructure. Since I was not on this project, I started running some tests hoping to re-produce the problem. This did not become too much of an issue as submitting as little as 8 messages to BizTalk would create some anomalies.

Anomaly #1 - Missing HAT Data

While trying to reproduce the problem, I noticed that we were missing some Tracking data related to HAT.

We would see instances that had started and that "apparently" had not completed. When stepping through the orchestration debugger there were some common shapes where the tracking would just stop.

However, within this process we have several log points where we will insert a log event into the event viewer. We also are communicating with a few different systems and were able to determine that these downstream systems were receiving messages from BizTalk.

Anomaly #2 - Missing BAM Data
Within the BAM Portal we would find columns that did not contain any information. We are using milestones, so the process was completing yet, there was data that was missing. Within this process we are using both the BAM API and TPE. TPE only tracks data that hits ports where as we had some information that never hits a port which forced us to use the BAM API.



What did not add up was that for an instance that was completed, and had missing data, the Expression shape that included the BAM API Call was being executed.

At this point I thought that the two anomalies could be related since we are taking advantage of the OrchestrationEventStream within the API. Darren Jefford has a great explanation of the OrchestrationEventStream here so I am not going to try and come up with something better here.

After some further investigation I was not getting too far so I figured it was time to reach out for some technical support. What I was told was that: "the TDDS sequence numbers in TDDS_SteamStatus tables and TrackingData tables have gotten out of sync." This will result in the TDDS subservice discarding Tracking data due to the synchronization issue. This definitely explains the missing data in HAT.

The previous description is more directly related to the missing HAT data. Since the BAM data is essentially Tracking data and following the same process as HAT data, due to the OrchestrationEventStream, it was plausible that both events were completely related.

Here are the steps that we performed to solve our issue:
(Disclaimer in effect - Please execute caution before running these steps. There is no way to be 100% sure that your issue was the same as mine. If you are unsure, contact Microsoft.)

• Stop all BizTalk Host Instances
• Stop SQL Agent (These two steps will ensure that no one is connecting to the database(s) )
• Ensure that you have no Active, Dehydrated, or ready to run instances in BizTalk Admin
• Verify that the TrackingData table has a row count of zero (You will need to drain any remaining messages by starting your tracking host back up, turn it off when row count gets to zero)
• Backup the Management, MsgBox, DTADb(Tracking), BAMPrimaryImport databases
• Truncate the following ables: BizTalkDTADB.TDDS_StreamStatus, BAMPrimaryImport.TDDS_StreamStatus

After performing these actions, we were set to run some tests to try and reproduce the issue. After running several scenarios and putting greater than production loads on the environment we still could not re-produce the issue.

What is still concerning about this issue is how it originally happened. There were no obvious changes to our environment that prompted this issue. We were also lucky that the event occurred in the the Test environment as rebuilding the environment was on the table at one point.

BizTalk 2006 - Message Viewer HAT Plugin

A tool that I have been using for a few months now, that I have not had a chance to blog about, is the Message Viewer HAT Plugin. The tool is available for free download on codeplex and was written/posted by someone with the handle of faizelk.

What does the tool do?
The tool allows you to view messages from within the Health and Activity Tracking (HAT) tool without having to save each message to disk.

Why should I use this tool?
If you have every played the "blame" game then you know that searching for messages inside of HAT can be a time consuming process. Especially if you don't have a promoted property to limit your search. You usually end up saving messages to disk to inspect them. This goes on for a bit until you actually find the message that you are looking for.

This tool will save you a lot of time as you don't have to leave the HAT tool to actually view message bodies. This of course assumes you have enabled message body tracking.

For those of you who are unfamiliar with the "blame" game that is when you are on an integration project with multiple parties and the finger pointing begins. HAT, with message body tracking enabled, becomes the tool of choice because you can provide "the evidence" of what BizTalk did. The conversation usually goes something like this: "BizTalk did what it was suppose to do, this is the message that I received and this is what I sent". The stakes and intensity of the game usually increase with the number of participants and systems involved. :-)

Install process
It is a simple process:

• Download the zip file from here


• Extract the Bizspoke.BizTalk.HATPlugIn.dll to the following BizTalk directory: X:\Program Files\Microsoft BizTalk Server 2006\Tracking\Plugins where 'X:' is the drive in which you have installed BizTalk on.

Running the Tool
The first thing you will need to do is launch HAT. Once HAT is available, click on "Queries" - "Most recent 100 service instances". Modify your query if desired.

Right mouse click on your desired message. You will now see a new item added to the menu. Click on the item called "(Bizspoke) View tracked messages...".

A .Net form will launch that will contain the "goods". From within this form you will find:

• The Message Body displayed on the right hand side


• The Message's context properties in the lower left


• The ability to Save the Message(s). This function works the same way that it does from within HAT; it will save all of the messages and context files


• The ability to view the XML file in Internet Explorer


• If you have selected a bunch of messages that you want to view, you can use the "Previous" and "Next" buttons to navigate through them.

Consider the scenario where you are tracking messages before and after port processing. These scenarios would include dealing with Flat Files where you have built a Flat File schema and are calling it in a Pipeline or when using the SAP Adapter to receive/send IDocs. In these scenarios, you have the ability to choose which message(s) that you want to track from within the Port - Tracking configuration in BizTalk Admin. If you choose to track both messages, the Message Viewer Plug-in will list both messages in the left frame and you can then choose which file you want displayed in the message body frame.

In my example, I have tracked a SAP IDoc(which is essentially a flat file) so it will get displayed in the "Text" tab. This message represents one that has been tracked before port processing (I received the message).

The second message body is the XML version of the IDoc; the one that BizTalk will use internally. By selecting this message and the "XML" tab, I can see it in a XML friendly format.

BizTalk Versions?
I have been using this tool with BizTalk Server 2006 R1. We have it deployed in all of our environments including Production - it is that valuable. If you have been keeping up with this blog, you may have noticed that I have a post regarding HAT's status in BizTalk 2009 Beta. For kicks, I copied the dll into the equivalent folder in my BizTalk 2009 installation folder and it didn't have any affect(it does not show up in any BizTalk Admin Console menus). So I am not sure how this tool will fit in moving forward, but if you are running BizTalk Server 2006, I recommend installing this tool.

BizTalk 2009 Beta - Where's my HAT?

The Health and Activity Tracking tool, otherwise known as HAT is no longer available in BizTalk 2009. Well at least not in its current Beta form. The HAT tool allowed BizTalk developers/admins the ability to view BizTalk's "processing logs". You could see when a message was received, when it was processed by an orchestration (if applicable) and when the message was sent to a downstream system.

You also had the ability to customize the default queries and save them if you wished. Provided you had message body tracking enabled, you could also save the message bodies and message context from HAT.

At least for now, you willl no longer see the HAT shortcut in your Windows Start Menu. I assume that the change is permanent but since this is beta you never know.



The HAT executable (BTSHatApp.exe) still exists in your BizTalk install folder (x:\Program Files\Microsoft BizTalk Server 2009), however you cannot launch it as it will provide an error.

Harold Hoffman was able to get the tool to launch from the command line by providing the right parameters but the tool itself doesn't really allow you to do much with it.

So with the disppearance of HAT, have we been hung out to dry? The simple answer is No! The HAT functionality that we have all grown to love and hate can now be found in the BizTalk Admin console. This functionality can be found from the Group Hub Overview page. To create a new Query click on the "New Query" tab.

Note: For the following images right mouse - > Open Link in New Window for a larger image


You can narrow your search by specifying the types of BizTalk Service instances that you are interested in. For instance if you were only interested in Suspended Services instance you could select this value from the "Value" drop - down.


If you are interested in finding a message that was previously received or sent then selecting "Tracked Message Events" from the drop - down would be a good choice. From here you can also save the message body, if message body tracking was enabled, by right mouse clicking on a result and selecting "Save to File".

If you want to further constrain your results, you can add criteria to your search. In my scenario I have added "Schema Name" to the "Field Name" column. When you click on the "Value" drop - down you will see a list of deployed schemas to select.

Dealing with adding dates into your HAT query was always a pain; you had to account for you local timezone when constructing your query. This either meant using the DateAdd function or manually doing the timezone calculation in your head. Nice to see that they have made this process smoother.

A calendar control is included which makes date restriction much easier than in the HAT custom query window.

As you can see in the image below, that I didn't have to make any accommodations for GMT and local timezones.


If you were looking for more than just messages (say an orchestration) you could do a search for "Tracked Service Instances". You can further restrict your results by including additional criteria from the "Field Name" drop - down. For instance if you could restrict your results to a particular BizTalk Host.

Within the BizTalk Admin Console, the product team has included some convenience features like the calendar control and pre-populating drop - down controls. I did notice an inconsistency between the Host Name drop downs when performing searches against "All In-Process Service Instances" and "Tracked Service Instances".

As you can see BizTalkServerApplication and BizTalkServerIsolatedHost have been pre-populated when performing searches for "All In-Process Service Instances".




In the case of "Tracked Service Instances" the Host Names are not pre-populated. When I did insert the name of my Host and clicked the "Run Query" button it did return my desired result. I am not sure if this is a bug or design decision but remember...its Beta. I have submitted this to the Microsoft Connect site and we will see what happens with it.



Conclusion
Moving the HAT functionality into the BizTalk Admin console was definitely the right decision. Having to use 2 different tools from BizTalk 2006 onward was a bit of a pain. The BizTalk Admin console was a HUGE step forward from BizTalk 2004 so to have all of this functionality in one tool is great. After using HAT for almost 5 years, it will take a little getting use to but I am looking forward to using the BizTalk Admin console for all of my historical tracking needs.

What version of BizTalk do I have?

I have seen this question asked several times in the forums and newsgroups so I figured it would be worth posting. I guess it relates more to determining whether you have BizTalk 2006 R1 or BizTalk 2006 R2 installed as BizTalk 2009 has some noticeable branding differences that R1 and R2 do not have.

The content itself comes from the BizTalk 2009 MSDN documentation, but I am posting for quick reference and in case the MSDN link changes(it is beta documentation).


To determine your version, do the following:

1. From RegEdit, navigate to the HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\BizTalk Server\3.0 key.


2. Check the ProductVersion key and note the version number.


3. Map the version number from your key to the table below:

3.0.4902.0 BizTalk Server 2004
3.0.6070.0 BizTalk Server 2004 SP1
3.0.7405.0 BizTalk Server 2004 SP2
3.5.1602.0 BizTalk Server 2006
3.6.1404.0 BizTalk Server 2006 R2

If the value of the ProductVersion key is greater than 3.6.1404.0, the version of BizTalk Server is BizTalk Server 2009.

On my machine I am running BizTalk 2009 Beta and my version is 3.8.104.5 but do remember that this is a Beta release and this version number is bound to change.

Microsoft MVP Awarded: Part 2

I was recently informed that I have received MVP status from Microsoft for the 2nd consecutive year. The MVP experience is a great one; it is a lot of hard work, but provides excellent opportunities to exchange information with Microsoft employees and other MVPs.

The highlight of my first year was definitely attending the MVP Summit and interacting with both the Product Group and other MVPs from all of the world. I met some great people there and continue to stay in regular contact with many of them.

If you would like to see my MVP profile, you may see it here.