BizTalk HotRod - Issue 5, Q1 2009 is now available

The latest version of BizTalk HotRod is available at http://biztalkhotrod.com/default.aspx. The issue is definitely worth reading and covers the following topics:

• BizTalk vs Dublin
• Unit Testing in BizTalk 2009
• BizTalk - Sharepoint integration
• WCF + BAM
• Operations Management for BizTalk using SCOM
• and much more

For those of you who are still unsure where BizTalk and Dublin fit in, the BizTalk vs Dublin is a much read. BizTalk and Dublin definitely have some similar features and capabilities. The author, Stephen Kaufman , provides good scenarios as to which technology stack should be used and when it should be used.

The Unit Testing article provides a great walk through of the new unit testing capabilities that are available in BizTalk 2009. Step by step examples are provided for Testing Schemas, Maps and pipelines.

BizTalk Adapter Pack v2 + SQL Query Notifications

I ran into a great post by Richard Seroter that discusses the new capabilities of Query Notifications in BizTalk 2009 when using the BizTalk Adapter Pack.

Query Notifications allow BizTalk to listen for changes that occur on the SQL Server that you are interested in receiving updates from. In previous versions of BizTalk you would usually attack this scenario with a Polling Receive location and then update the records that you have just processed or store a timestamp of the last time that you polled. This would allow you to only process new records.

Polling is nothing new to BizTalk scenarios. Many adapters support polling including POP3, FTP, File and Database adapters. Other adapters will listen for events, or messages to be raised and then BizTalk will process the message. These adapters include SAP, SOAP and HTTP.

In many Database scenarios you may poll every minute and only retreive information periodically. No data may have changed since the last polling attempt. This is wasting resources as you are constantly connecting to check if there is any work for BizTalk. By using this Event driven model, BizTalk will receive a notification that there is work for it, but the initial notification will not contain the data that is changed. BizTalk will then need to make a connection to see what records have changed.

BizTalk will still need to manage which records have been processed. However, BizTalk is able to execute a Select statement to find the updated records and update them all in the same transaction so that they will not get processed again. See Richard's post for more details.

So if you have not had a chance to walk through Richard's blog post I highly reccommend that you do. Not having used the SQL Service broker before I did have a few problems when I started up my sample.

The Messaging Engine failed to add a receive location "WcfReceiveLocation_SqlAdapterBinding_Notification_Custom" with URL "mssql://btsserver//BizTalkSQLNotify?" to the adapter "WCF-Custom". Reason: "System.InvalidOperationException: The ChannelDispatcher at 'mssql://btsserver//BizTalkSQLNotify?' with contract(s) '"IOneWayAsyncTxn"' is unable to open its IChannelListener. ---> System.InvalidOperationException: The SQL Server Service Broker for the current database is not enabled, and as a result query notifications are not supported. Please enable the Service Broker for this database if you wish to use notifications.

If you receive the preceding error you can follow these steps to enable your Service Broker for the database that you want to listen for notifications. Run the following commands (at your own risk) in SQL Server Managment Studio

SELECT is_broker_enabled FROM sys.databases WHERE name = 'your_database_name'

If the value returned is 0 then your broker is currently disabled. Continue with the following steps:

ALTER DATABASE your_database_name SET NEW_BROKER WITH ROLLBACK IMMEDIATE

ALTER DATABASE your_database_name SET ENABLE_BROKER

You should now be able to run this SQL Statment and have the result be '1'. This will indicate that the broker is in fact enabled.

SELECT is_broker_enabled FROM sys.databases WHERE name = 'your_database_name'

The next error that I received was this one:

Service Broker needs to access the master key in the database ''. Error code:26. The master key has to exist and the service master key encryption is required.

To resolve this issue I rant the following command:

CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'your_password'

This password can be anything, you it is not your SQL Server password.

BizTalk Adapter 3.0 vs SAP Enterprise Services

For those of you who are interested in integration with SAP, there is a document worth reading called Positioning SAP Enterprise Services vs Microsoft BizTalk Adapter 3.0 for mySAP Business Suite. The document itself describes the different scenarios in which you can use the BizTalk Adapter Pack 3.0 when integrating with SAP. These scenarios include:

• .Net Based Applications
• SQL Server 2005 Integration Services
• SQL Server 2005 Reporting Services
• Sharepoint
• and of course BizTalk.

In case you are not aware, the BizTalk Adapter packs are based upon WCF and therefore can be run outside of BizTalk even though they are branded "BizTalk". The Adapter pack itself is included in the license of BizTalk, but you cannot run the Adapter pack on a server without BizTalk unless you have a separate license for it or you do not have BizTalk installed. See the Adapter pack pricing page for more licensing info.

In my last post, I discussed how the SOAP and SQL Adapters have been deprecated for BizTalk 2009. Well, you can add another adapter to the list and that is the Microsoft BizTalk Adapter v2.0 for mySAP Business Suite. This is the adapter that originated in BizTalk 2006 R1 on the Microsoft BizTalk Adapters for Enterprise Applications cd.

The primary objective of this document is to discuss the scenarios where you would want to use the BizTalk adapter versus when you would want to use the Web Service capability of Enterprise Services. SAP ECC 6.0 natively provides Web Service capabilities. So this document is probably to address those opinions that if SAP has Web Services then why do I want to use BizTalk,or the Adapter pack, to exchange data between SAP and another application?

Here are a couple of ideas that I can think of:

• Avoid point to point integration scenarios
• Take advantage of BizTalk's transformation(mapping) capabilities
• Durable messaging (retries, persistence)
• Monitoring(MOM,SCOM)/Tracking(HAT,BAM)
• Managing a Business Process between multiple systems.
• Adapter provides reliable, performant and secure data transmissions.

BizTalk 2009 Beta Documentation - now available

The BizTalk 2009 Beta Documentation is available here. (Thanks Mick Breeze / Rahul Garg ).

While 'flipping' through the documentation I discovered an interesting section: Deprecated Features, Tools, and APIs. Within this section, the following features have been marked as deprecated:

• BizTalk Explorer
• SOAP Adapter
• SQL Adapter

For those of you who may be new to BizTalk, BizTalk Explorer is a view that is available inside Visual Studio that allows you to connect to your BizTalk Management database. Personally, I have not used this view since BizTalk 2004, but there may be some out there that started using it in 2004 and never let it go.

SOAP and SQL Server connectivity will still be supported in the future by BizTalk, but by using a different adapter. Basic Profile Web Services are a supported scenario in the base WCF adapter so any integration with basic Web Services should be done using the WCF-BasicHttp adapter instead of the SOAP adapter.

The BizTalk Server Adapter pack 2.0 includes a sqlBinding that allows you to communicate with SQL Servers using the WCF adapter. Currently, the supported versions of SQL that you can communicate with include SQL 2000, 2005 and 2008.

The guidance around deprecated features is that you should not use them when developing new applications. This also gives you a transition period to move off of these features before the next release of BizTalk (post 2009).

A bi-product of these features being deprecated are the tools that compliment these features. The tools being deprecated include:

• BizTalk Web Services Publishing Wizard
• Add Web Reference

Both of these tools have their equivalent WCF versions so you will use these exclusively in the future.

Within the BizTalk 2009 release, we have seen a few features removed including MSMQT and HAT that had been previously deprecated. So we know that Microsoft means business when they provide these early warnings. So my advice to you would be to start using the WCF adapters where possible so that you are not re-factoring your apps at the last minute in the future.

BizTalk 2009 Beta - Where's my HAT?

The Health and Activity Tracking tool, otherwise known as HAT is no longer available in BizTalk 2009. Well at least not in its current Beta form. The HAT tool allowed BizTalk developers/admins the ability to view BizTalk's "processing logs". You could see when a message was received, when it was processed by an orchestration (if applicable) and when the message was sent to a downstream system.

You also had the ability to customize the default queries and save them if you wished. Provided you had message body tracking enabled, you could also save the message bodies and message context from HAT.

At least for now, you willl no longer see the HAT shortcut in your Windows Start Menu. I assume that the change is permanent but since this is beta you never know.



The HAT executable (BTSHatApp.exe) still exists in your BizTalk install folder (x:\Program Files\Microsoft BizTalk Server 2009), however you cannot launch it as it will provide an error.

Harold Hoffman was able to get the tool to launch from the command line by providing the right parameters but the tool itself doesn't really allow you to do much with it.

So with the disppearance of HAT, have we been hung out to dry? The simple answer is No! The HAT functionality that we have all grown to love and hate can now be found in the BizTalk Admin console. This functionality can be found from the Group Hub Overview page. To create a new Query click on the "New Query" tab.

Note: For the following images right mouse - > Open Link in New Window for a larger image


You can narrow your search by specifying the types of BizTalk Service instances that you are interested in. For instance if you were only interested in Suspended Services instance you could select this value from the "Value" drop - down.


If you are interested in finding a message that was previously received or sent then selecting "Tracked Message Events" from the drop - down would be a good choice. From here you can also save the message body, if message body tracking was enabled, by right mouse clicking on a result and selecting "Save to File".

If you want to further constrain your results, you can add criteria to your search. In my scenario I have added "Schema Name" to the "Field Name" column. When you click on the "Value" drop - down you will see a list of deployed schemas to select.

Dealing with adding dates into your HAT query was always a pain; you had to account for you local timezone when constructing your query. This either meant using the DateAdd function or manually doing the timezone calculation in your head. Nice to see that they have made this process smoother.

A calendar control is included which makes date restriction much easier than in the HAT custom query window.

As you can see in the image below, that I didn't have to make any accommodations for GMT and local timezones.


If you were looking for more than just messages (say an orchestration) you could do a search for "Tracked Service Instances". You can further restrict your results by including additional criteria from the "Field Name" drop - down. For instance if you could restrict your results to a particular BizTalk Host.

Within the BizTalk Admin Console, the product team has included some convenience features like the calendar control and pre-populating drop - down controls. I did notice an inconsistency between the Host Name drop downs when performing searches against "All In-Process Service Instances" and "Tracked Service Instances".

As you can see BizTalkServerApplication and BizTalkServerIsolatedHost have been pre-populated when performing searches for "All In-Process Service Instances".




In the case of "Tracked Service Instances" the Host Names are not pre-populated. When I did insert the name of my Host and clicked the "Run Query" button it did return my desired result. I am not sure if this is a bug or design decision but remember...its Beta. I have submitted this to the Microsoft Connect site and we will see what happens with it.



Conclusion
Moving the HAT functionality into the BizTalk Admin console was definitely the right decision. Having to use 2 different tools from BizTalk 2006 onward was a bit of a pain. The BizTalk Admin console was a HUGE step forward from BizTalk 2004 so to have all of this functionality in one tool is great. After using HAT for almost 5 years, it will take a little getting use to but I am looking forward to using the BizTalk Admin console for all of my historical tracking needs.

BizTalk 2009 Beta - Code Migration Process

After going through the process of migrating a BizTalk 2006 R1 solution to BizTalk Server 2009 (beta) I am happy to report that the process is a painless endeavor. The process itself reminds me a lot of the BizTalk 2004 - 2006 migration process.

Scenario
My dev box has a clean install of BizTalk Server 2009 and did not previously have Visual Studio.Net 2005 on it. Therefore, I needed to go through the Code Conversion process. I used a BizTalk 2006 project from a previous blog post. The project was not all that complex but it did include the typical artifacts that you would find in a BizTalk solution. This includes:

• The BizTalk Solution file
• Orchestration Project
• MapsAndSchema Project
• Pipeline Project
• .Net Helper Project
• Binding Files (more on this later)

Process
As you can see the DeliveryNotification sln file is a Visual Studio '8' (or VS.net 2005) file.

Double clicking on this file will launch Visual Studio 2008 and the Code Conversion Wizard

After clicking on "Next" you will be prompted with a dialog box that lists all of the related projects that will get migrated.

Success! Enable the check box to see the conversion report.

The results of the conversion process are documented below and the project listings are on the right hand side. Note that BizTalk projects currently have a "C#" icon. I do expect this to change as this is Beta. The underlying reason for this is that BizTalk projects are now considered to be 'flavours' of C# projects as discussed in this post.

So what was converted?
Here is a screenshot of what my BizTalk artifacts look like after the conversion process. By observing the Date modified timestamps, you get a sense of what was changed.

After taking a closer look, you will find that the actual BizTalk artifacts(schemas, orchestrations, maps etc) are not actually getting modified. Rather, it is the Visual Studio artifacts(solution file, project file, user settings) that are being modified.

I look at this as good news/bad news. Good in the sense that you will have a lot confidence that nothing will get broken when converting code. Bad in the sense that this means little has changed(from a development perspective) between BizTalk Server 2006 Rx and BizTalk Server 2009

Post Conversion
So the conversion process worked - now what? When I tried to build and deploy the solution I had no issues:

When I launched the BizTalk Admin Console, I was pleased to see that my Application Container and related artifacts were created during my deployment from VS.net 2008. Since this was my initial deploy on a clean dev box, I didn't have any bindings for the orchestration, receive locations or send ports.

I decided to import an existing binding file from my BizTalk Server 2006 solution.

The binding file import was successful!

I did discover an issue when I tried to start up the application from the tree view on the left hand side:

I could not Start, Stop or Configure the application. I was able to manually start the send ports, the orchestration and enable the receive locations. Or I could click on "Applications" folder and start the application from the "main frame". Both of these actions did start the application and turned the application's icon green.

I checked the Microsoft Connect site and saw that Ben Cline had already reported this as a bug. I added some of my own comments so Microsoft is aware of it and I would expect that it will get resolved for the next release.

Once the application was online, I processed some files to ensure that the application was functioning correctly and that it was.

Conclusion
While my project was not all that complex, I was happy with the migration experience and feel pretty confident in moving my projects from BizTalk 2006 to BizTalk 2009.