BizTalk vNext Wish list

So you are probably thinking, BizTalk 2009 isn't even out and you are already posting a wish list? Well yes, I was recently asked the question, so I figured I would share my thoughts with you.

Adapters

• An Email Adapter(POP3/Exchange) that supports regular expressions. For instance if you are only interested in retrieving certain messages from a mailbox, you should be able to provide a regular expression against the subject field of the message. If the words in the subject matches you reg ex, then the message would be retrieved.


• A native Exchange adapter. I have personally run into this and have also seen several instances of this on the MSDN forms. For many organizations, leaving their servers open to POP3 connections posses a security risk. The way that we solved this problem was limiting POP3 access to certain IP Addresses. This is "ok" for servers, but when I put in a request to have my IP Address added, that was declined as individual users should not have POP3 access. So yes this is more of "my problem" than Microsoft's, but if many others are in the same problem then perhaps adding an Exchange adapter would be beneficial.


• FILE Adapter that supports regular expressions. The current adapter does support file masks using '*', '?' etc, but it would be even better if regular expressions were supported. I work in an industry where the names of files are built into the File Exchange specification. Often times this will include ranges of numbers that are considered valid. It is pretty tough to properly constrain without regular expressions. Yes, I know that a Custom Adapter could be built that supports this scenario, however it would be nice if this was included in the core product.


• FTP Adapter that supports Temporary folder in ASCII mode. In order for the FTP adapter to support "once only guaranteed delivery", you are required to use a Temporary folder when moving files with the FTP Adapter as described here. The problem is that temporary files are only supported when the Adapter is used in binary mode. The issue is that when are exchanging data between heterogeneous environments like Windows and Unix that these environments use different symbols for carriage return line feeds. Using ASCII mode takes care of the conversions of these symbols. So as you can see we are in a bit of a Catch 22 situation here.

Deployment

• Deployment has improved considerably from BizTalk 2004, but I still think that some improvements could be made. The organization that I work for has automated our build and deploy process including deploying a BizTalk project to a multi-node BizTalk group. It would be nice if this type of functionality was part of the core product. For instance when importing an MSI in the BizTalk Admin console if you could "Deploy to Group" that would be a welcomed feature where it does the import to the BizTalkMgmt database once and gac's the dlls on the remaining BizTalk nodes.


• Binding files - To be honest I am not exactly sure what I am looking for here but think there just has to be a better way. Obviously the ability to separate code/assemblies from configuration is required. However managing that configuration could be improved. If you have a project with only a few receive locations then binding files is not too big of a deal. However once you start reaching the 10+ receive locations managing this data and the passwords becomes tedious. I personally 'love' having to export the bindings after a deployment only to play the "Find and Replace" game with the password '*'s before checking the bindings into source control. I do this so that the next time I deploy I don't have to manually set passwords during the next deployment. I do recognize that having passwords, in binding files, in clear text is not a great option but if we could somehow improve this scenario it would relieve a lot of headaches.

Design Time

• The ability to update a generated schema automagically. For instance if I have generated a schema for an SAP IDoc and the IDoc has now changed, I need to run through the "Add Generated Items" wizard again. If I have renamed or modified the target namespace of the original schema I then need to update this new version of the schema. As an added bonus, a blank orchestration is added to my solution even though I already have a working solution.


• The ability to debug an orchestration from Visual Studio. When I first started using BizTalk, I really missed this feature. Prior to working with BizTalk I was a ASP.Net developer and really enjoyed ASP.Net debugging. Especially having worked with classic ASP. I am not sure how they could support this feature, but it would definitely be welcomed.

Management/Operations

• The ability to go back and historically look at a message. So yes this functionality does exist today however there is another catch 22 situation. If you have message body tracking enabled, then a copy of the message body will placed in the BizTalkDTADb database. However, in order to keep your BizTalk environment performing well, you need to archive and purge data from this database. The job that takes care of this is the DTA Purge and Archive job. If you neglect to enable this job, or keep your live window open for too long, you are bound to have performance problems. At one point, neglecting this job was one of the top BizTalk tickets to Microsoft support. One option is to take the extracts that this job will output and aggregate them together to create your own Long Term Archive solution. This would alleviate you from any run time issues, however it leaves you with a management problem as you need to manage this yourself. The requirement itself comes from a request to find out "What happened last month to order # 1234567"?


• If you have a complex business scenario and need to find all messages related to that instance of the business process, it would be great if you could link all of the related messages to the orchestration that managed this business process. So yes, there are Ids that do link all of these interactions together, but it would be nice if you could easily view all of this information from a tool. This would allow you to quickly view what happened to that particular business process instance. Consider the following interaction, if you needed to view all of the messages for this particular business process, how could you easily achieve this?

ETC

So here are a few items that tend to show up frequently elsewhere

• Support for Low Latency scenarios. BizTalk's design includes built in persistence. This is a feature that is used to support guaranteed delivery. While in many scenarios this feature is required, but there is a cost associated. For some scenarios this cost is just too expensive. Having the ability to by pass all of the persistence may satisfy some requirements that are not currently addressed.
• An expandable, or larger, expression editor. The argument has always been that if you need a larger expression editor then chances are that you are doing something that you shouldn't be doing. While I agree in principle with this statement, here is something else to consider. If you follow a popular namespace convention that includes using OrganizationName.BusinessUnit.FunctionalName as a namespace in a .Net assembly it does not take too long before you are starting to scroll to the right when calling a static method.
• Constructing a new message in an orchestration. We have all been there! You use some sort of questionable approach to create an instance of a message. This may include using a map that does not actually map any data, loading an XML string that matches your schema's format and assigning it to a message. It would be nice to just create a new instance of a message without having to jump through these hacks.

If you have any Wish list items of your own, please post them using the comments feature.

WindowsITPro Magazine - Get a Handle on Windows Performance Analysis

I recently had an opportunity to work with Michael Morales from Microsoft on an article for WindowsITPro magazine. The article focused on Performance Monitoring and my contribution was primarily focused on PAL (Performance Analysis of Logs). If you are unfamiliar with PAL you can read more about it here.

If you are interested in reading the article it is in the February 2009 edition of WindowsIT Pro magazine or you can view the article online here.

BizTalk HotRod - Issue 5, Q1 2009 is now available

The latest version of BizTalk HotRod is available at http://biztalkhotrod.com/default.aspx. The issue is definitely worth reading and covers the following topics:

• BizTalk vs Dublin
• Unit Testing in BizTalk 2009
• BizTalk - Sharepoint integration
• WCF + BAM
• Operations Management for BizTalk using SCOM
• and much more

For those of you who are still unsure where BizTalk and Dublin fit in, the BizTalk vs Dublin is a much read. BizTalk and Dublin definitely have some similar features and capabilities. The author, Stephen Kaufman , provides good scenarios as to which technology stack should be used and when it should be used.

The Unit Testing article provides a great walk through of the new unit testing capabilities that are available in BizTalk 2009. Step by step examples are provided for Testing Schemas, Maps and pipelines.

BizTalk Adapter Pack v2 + SQL Query Notifications

I ran into a great post by Richard Seroter that discusses the new capabilities of Query Notifications in BizTalk 2009 when using the BizTalk Adapter Pack.

Query Notifications allow BizTalk to listen for changes that occur on the SQL Server that you are interested in receiving updates from. In previous versions of BizTalk you would usually attack this scenario with a Polling Receive location and then update the records that you have just processed or store a timestamp of the last time that you polled. This would allow you to only process new records.

Polling is nothing new to BizTalk scenarios. Many adapters support polling including POP3, FTP, File and Database adapters. Other adapters will listen for events, or messages to be raised and then BizTalk will process the message. These adapters include SAP, SOAP and HTTP.

In many Database scenarios you may poll every minute and only retreive information periodically. No data may have changed since the last polling attempt. This is wasting resources as you are constantly connecting to check if there is any work for BizTalk. By using this Event driven model, BizTalk will receive a notification that there is work for it, but the initial notification will not contain the data that is changed. BizTalk will then need to make a connection to see what records have changed.

BizTalk will still need to manage which records have been processed. However, BizTalk is able to execute a Select statement to find the updated records and update them all in the same transaction so that they will not get processed again. See Richard's post for more details.

So if you have not had a chance to walk through Richard's blog post I highly reccommend that you do. Not having used the SQL Service broker before I did have a few problems when I started up my sample.

The Messaging Engine failed to add a receive location "WcfReceiveLocation_SqlAdapterBinding_Notification_Custom" with URL "mssql://btsserver//BizTalkSQLNotify?" to the adapter "WCF-Custom". Reason: "System.InvalidOperationException: The ChannelDispatcher at 'mssql://btsserver//BizTalkSQLNotify?' with contract(s) '"IOneWayAsyncTxn"' is unable to open its IChannelListener. ---> System.InvalidOperationException: The SQL Server Service Broker for the current database is not enabled, and as a result query notifications are not supported. Please enable the Service Broker for this database if you wish to use notifications.

If you receive the preceding error you can follow these steps to enable your Service Broker for the database that you want to listen for notifications. Run the following commands (at your own risk) in SQL Server Managment Studio

SELECT is_broker_enabled FROM sys.databases WHERE name = 'your_database_name'

If the value returned is 0 then your broker is currently disabled. Continue with the following steps:

ALTER DATABASE your_database_name SET NEW_BROKER WITH ROLLBACK IMMEDIATE

ALTER DATABASE your_database_name SET ENABLE_BROKER

You should now be able to run this SQL Statment and have the result be '1'. This will indicate that the broker is in fact enabled.

SELECT is_broker_enabled FROM sys.databases WHERE name = 'your_database_name'

The next error that I received was this one:

Service Broker needs to access the master key in the database ''. Error code:26. The master key has to exist and the service master key encryption is required.

To resolve this issue I rant the following command:

CREATE MASTER KEY ENCRYPTION BY PASSWORD = 'your_password'

This password can be anything, you it is not your SQL Server password.

BizSpark - Calling Startups

I recently ran across an interesting program from Microsoft called "BizSpark". (No that is not BizTalk spelled incorrectly).

For many start ups, initial software licensing is a considerable investment and in many cases an obstacle. For many, a way around this obstacle is to look to the LAMP stack. LAMP stands for Linux, Apache, MySQL and PHP (or Perl, Python etc). These technologies are either free and open source. This post is not intended to slam either approach or discuss the strengths or weaknesses of each approach, but rather provide information about the BizSpark program.

Since Microsoft software costs money, many people will take the LAMP route to avoid up front software costs. For many businesses, they would be happy to pay the license fees if they knew that their business would be successful and profitable. But what if there was a way to use Microsoft Software for a period of time without having to pay license fees. Would that change your mind to go the LAMP route? If so then this program may be for you.

The BizSpark program is designed to allow start ups the opportunity to use some Microsoft software in Development and Production environments for free for a period of up to 3 years provided you meet their criteria. This criteria includes:

• Is in the business of software development
• Is privately held
• Has been in business for less than 3 years, and
• Has less than US $1 million in annual revenue

Please refer to the actual BizSpark site for the fine print.

So what software is available?

For development, testing and demonstration: –
The software included in the Visual Studio Team System Team Suite (VSTS) with MSDN Premium, plus VSTS Team Foundation Server (Standard Edition) - for the entire development team

For production use to deploy and host Startup's new "software as a service" application to be delivered over the Internet to Startup's customers:
Windows Server; SQL Server, BizTalk Server, and Office SharePoint Server for Internet Sites hosting, and – Systems Center for managing hosting server operations.

As you can see the software available in the program is some of their flagship (and more costly) software. Even though BizTalk is considerable more cost effective than its competitors, a common complaint is that it is a considerable upfront investment to pay for the Windows Server, SQL Server and BizTalk licenses. If you could use this software for nothing while your business is in its infancy, it could be a great opportunity to use good tools and not have to worry about licencing costs.

So obviously not every organization would qualify for this program, but for those in start-up mode it could be a great opportunity. Also see the Program Guide for more information.