4 questions...with me

This month I am featured in Richard Seroter's 4 questions series. I am a big fan of his blog and this feature so I had no hestitation when it came to accepting his invitation. Here is the link to the interview: http://seroter.wordpress.com/2009/08/03/interview-series-four-questions-with-kent-weare/

BizTalk 2009: NSoftware Adapters 3.5.3477

UPDATE: Note this post has been updated see below.

There are some new features that are part of the nSoftware Adapter pack version 3.5.3477 that was recently released. I have blogged about these adapters before and continue to use them at my ogranization so I figured I would post an update.

What adapters are included in this bundle?
AS2 Adapter - eBusinessReady Certified AS2 (EDI-INT) Adapter.
FTPS Adapter - SSL-Enabled Secure FTP Adapter for BizTalk.
SFTP Adapter - Secure Shell (SSH) Enabled FTP Adapter for BizTalk.
OFTP Adapter - Odette File Transfer Adapter for BizTalk.
Credit Card Adapter - Credit card transaction processing Adapter.
RSS Adapter - RSS Client integration Adapter.
Secure Shell (SSH) Adapter - Secure Shell Remote Execution (SExec) Adapter.
Secure Email Adapters - Secure Email (SMTP/POP/IMAP) Adapter.
SMPP Adapter - Short Message Peer-to-Peer SMS messaging Adapter.
SNPP Adapter - Alphanumeric paging Adapter.
XMPP Adapter - Jabber / XMPP Instant Messaging Adapter.
S3 Adapter - Integration with Amazon Simple Storage Service (S3).
Zip Pipeline Component - PKZip/Winzip compatible Zip compression/decompression.
Gzip Pipeline Component - UNIX compatible (LZW/LZC) Gzip compression.
ACH Pipeline Component - Electronic Check (ACH) message processing.
PGP Pipeline Component - PGP Sign, Verify, Encrypt, and Decrypt.

What about BizTalk 2009?
BizTalk 2009 is fully supported

64bit support?
Yes, unlike the Microsoft out of box adapter you do not have to run this FTP adapter in a 32 bit host.

What is new with the adapters?
Support for temporary folders now exists in the S/FTP adapter. In the previous versions of the adapter they supported a temporary file extension. They now support a temporary folder. This temporary folder is an important feature due to the reliability concerns involving FTP. The purpose of this folder is that you write the entire contents to this dirctory and once all contents have been delivered, a command is executed(under the hood) that will move the file from the temp folder to the destination folder. The tempory file extension had a similar purpose but if a down stream system is only looking at the first portion of the file name it won't relize that it has a temporary file extension until it is too late. The Microsoft FTP adapter also supports the notion of a temporary folder, however it is only available in Binary mode. This doesn't help out in heterogeneous environments (Win and Unix) where CRLF characters mean something different to each system.

All adapters now support Dynamic and Static modes. Believe it or not previous versions of the adapters never fully supported a dynamic mode. I previously blogged about how you could make a static port "dynamic like" by overriding certain context properties, but you still had to have a Static Send Port.

I ran into a couple gotchas while testing out this new dynamic ability so I figured that I would put together a post. Here I have a very simple solution that will receive a file via the File Adapter. I then copy this inbound message to an outbound message of the same type, update a few context properties, set the dynamic URI in an expression shape and then send the message out through a logical dynamic Send Port.

In order to access the nSoftware context properties in the message assignment shape, I needed to add a reference to my BizTalk project. You will find this Dll in the Adapter's install lib directory.

You can override pretty much any context property that you can think of:

I was only interested in setting a few properties to keep this post simple. I set the following properties:

• User (Username)
• Passsword
• SSLStartMode
• Override

The SSLStartMode property is important. Technically this adapter is an SFTP adapter and if you choose not to use SSL then it basically becomes a FTP adapter. The default is to have SSL enabled so I have decided to turn it off.

The next thing that I need to do is set the URI for this dynamic logical port. If you look closely you will notice that my URI contains the "ftp" protocal identifier. So how does BizTalk know to use the NSoftware adapter or the Microsoft adapter? No, you don't have to uninstall the Microsoft adapter.

A context property that was new to me is the
Microsoft.XLANGs.BaseTypes.TransportType property. This property must have shown up in BizTalk 2006 R2 with the introduction of the WCF adapter. A similar situation would otherwise happen since indicating "http" in your dynamic URI would be a cue for the HTTP adapter to send the message. If you go into the BizTalk admin console and down to Platform Settings -> Adapters you will see that the nSoftware adapter is referred to as
"nsoftware.FTP v3".

I initially ran this project against a Windows environment but then had some problems with connecting to a Unix server. I am not so sure that it is a Unix issue per say but perhaps the way we mount different disks to our Unix Servers? In any event I had to make a few changes to the context properties that I set in the Message Assignment Shape:

• I populated the Remote path with the full directory structure
• Set the RemoteFile property with the name of the file that I wanted to send

In the Expression shape where I populate the dynamic URI, I had to provide less information. I only need to be concerned with providing the server name as the folder and file name information has been populated in the context properties.

***UPDATE***

The "abnormal" situation that I encountered with the Unix environment has been fixed in version 3.5.3488. You can obtain this release from nSoftware support support@nsoftware.com

***************

So why the nSoftware adapter?
I think most people who have chosen the nSoftware adapters have done so primarily for their SFTP and FTPS adapters. Out of the box, Microsoft does not provide an FTP adapter that supports security. Meanwhile FTP remains a popular protocal when exchanging information with internal and external systems. When connecting with external systems, usually some sort of security is required. There is also a SFTP adapter on codeplex. I have never used it, but it has some pretty good reviews. I would not be able to introduce this adapter into my environment due to it being open source which leaves me to look at a 3rd party like nSoftware.

The ability to use a temporary folder or file extension may also be enough for some customers who have a mixture of environments that they communicate with. FTP is not all that reliable as it is so having these features provides you with a little more durability.

BizTalk Light and Easy Viewing Series - SAP Adapter

A couple months back Mick Badran, a BizTalk MVP from Australia, got a few MVPs and other BizTalk experts together to put on a series of webcasts that cover various topics in BizTalk Server 2009.One of the topics that I decided to tackle was the new SAP Adapter ( as found in BizTalk Adapter Pack). In this webcast I cover the following topics:

• SAP Background Information
• Why do companies use SAP
• Challenges with integrating with SAP
• Previous SAP Adapters
• BizTalk Adapter Pack 2.0
• Demos

Alan Smith has launched a site that will host the Light and Easy series in addition to other Microsoft related webcasts. If you are interested in seeing this webcast here is the link: http://www.cloudcasts.net/ViewWebcast.aspx?webcastid=2521554266599869567

If you have any questions or comments about the webcast, feel free to leave a comment on the blog.

BizTalk Light and Easy Viewing Webcast Series - High Availability with BizTalk Server 2009

A couple months back Mick Badran, a BizTalk MVP from Australia, got a few MVPs and other BizTalk experts together to put on a series of webcasts that cover various topics in BizTalk Server 2009.

One of the topics that I decided to tackle was High Availability in BizTalk Server 2009. In this webcast I cover the following topics:

• High Availability - What is it?
• How do I achieve High Availability with BizTalk Server?
• Impact on Adapters and Hosts
• Impact on Databases
• Role of Enterprise Single Sign On
• Deployment scenarios
• Changes for BizTalk 2009

Alan Smith has launched a site that will host the Light and Easy series in addition to other Microsoft related webcasts. If you are interested in seeing this webcast here is the link. http://www.cloudcasts.net/ViewWebcast.aspx?webcastid=2521554265716170751

If you have any questions or comments about the webcast, feel free to leave a comment on the blog.

Microsoft Tech Days Canada is back for 2009

Microsoft Tech Days 2009 is coming back to Canada this fall to the following cities:

• Vancouver (September 14-15)
• Toronto (September 29-30)
• Halifax (November 2-3)
• Calgary (November 17-18)
• Montreal (December 2-3)
• Ottawa (December 9-10)
• Winnipeg (December 15-16)

Looks like Riderville (Regina) is off the list this year if you are from Saskatchewan you will want to head to Calgary or Winnipeg.

There is currently an early bird special special of $299 which is a savings of $300. I went last year and you can't go wrong with a Microsoft conference at this price point. The session agendas include a wide variety of topics so whether you are a hardcore developer or IT Pro you are bound to learn something new at this conference.

Follow this link for more details.

Clustering MQ Series and BizTalk Send/Receive

We have a 3rd party application that uses MQ Series as an integration bridge. BizTalk is used to manage a business process and perform message transformation between this system and our ERP.

We have been using this configuration for the past couple years and it came time to upgrade this 3rd party application. This application now supports Websphere MQ 6.0 where as we were previously running Websphere MQ 5.3. Since it was time for an upgrade and we do have a pretty large BizTalk farm with a team that is responsible for supporting both BizTalk and MQ we decided to install MQ 6.0 on this same cluster(as BizTalk)...for better or worse(it is supported).

In our previous configuration we had MQ 5.3 running on an Active/Passive Cluster that did not include BizTalk. When running BizTalk 2006 and wanting to use the MQ Series Adapter, there is a component that you must install on the MQ Series servers. The component that is installed is called MQSAgent*. Now if you are running BizTalk 2004 then the component is called MQSAgent where as if you are running BizTalk 2006 it is called MQSAgent2. According to this doc, the component is still called MQSAgent2 for BizTalk 2009. In our previous configuration, I must admit it was rock solid. We never had any fail over issues whatsoever.

We recently ran into some issues in the Test environment with the new configuration. We found that you needed to ensure that MS Distributed Transaction Coordinator (MS DTC) and MQ Series on the same node in order for it to function correctly. MSDTC is leveraged to support Guaranteed Reliable message delivery.

The specific problem that we ran into was that when we failed the MQ Resource group over to a new node, BizTalk would essentially just hang(with respect to MQ message processing). We couldn't send or receive messages with MQ. I then ran into this kb: The BizTalk Server Adapter for MQSeries version 2.0 no longer retrieves messages from a clustered MQSeries queue manager when the queue manager fails over to a different cluster node. This paragraph accurately describes our scenario:

You may configure the Microsoft BizTalk Server Adapter for MQSeries version 2.0 to receive messages from a clustered MQSeries queue manager. If the queue manager fails over to a different cluster node, the BizTalk Server Adapter for MQSeries no longer retrieves messages from the clustered MQSeries queue. When this behavior occurs, the following event is logged in the Application event log:

Event Type: Warning
Event Source: BizTalk Server 2006
Event Category: BizTalk Server 2006
Event ID: 5740
Date: 6/20/2009Time: 10:16:40 AM
User: N/A
Computer:
Description:The adapter "MQSeries" raised an error message. Details "Error encountered on opening Queue Manager name = ISVC.MSG4.QM.T Reason code = 2059.".
For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

Later on in the kb it describes a "Workaround" *cough* HACK */cough* that is suppose to terminate the MQSAgent from the node that previously was hosting the MQ/MSDTC Resource group.

Option Explicit
On Error Resume Next

Dim sComputerName, oWMIService, colRunningServices, oService, colProcessList, objProcess

If Wscript.Arguments.Count = 0 Then

 sComputerName = "."
 Call ServStat
 Wscript.Quit
End If

Sub ServStat
Set oWMIService  = GetObject("winmgmts:" _
 & "{impersonationLevel=impersonate}!\\" & sComputerName& "\root\cimv2")
Set colRunningServices = oWMIService.ExecQuery _
 ("Select * from Win32_Service where DisplayName='Distributed Transaction Coordinator'")

For Each oService in colRunningServices
 'Wscript.Echo oService.DisplayName  & VbTab & oService.State
 if (oService.State="Stopped") Then
  'Wscript.Echo "Stopped"
  ' find the dllhost
  Set colProcessList = oWMIService.ExecQuery ("SELECT * FROM Win32_Process WHERE Name = 'DLLHOST.EXE'")

  For Each objProcess in colProcessList
   if inStr(objProcess.CommandLine, "6D06157A-730B-4CB3-BD11-D48AC6B8A4BB")>0 then

    'Wscript.Echo objProcess.ProcessId
    Dim objShell
    Set objShell = CreateObject("WScript.Shell")
    objShell.Run "cmd /k kill -f " & objProcess.ProcessId & "& exit"
    WScript.Quit
   end if
  Next
 end if
Next

End Sub

I have highlighted two issues with this script in red:

1. The ID 6D06157A-730B-4CB3-BD11-D48AC6B8A4BB that they are referring to in the script is for the MQSAgent (BizTalk 2004) and not BizTalk 2006. The MQSAgent2 ID (BizTalk 2006) is C691D827-19A0-42E2-B5E8-2892401481F5.
   
   



2. The other issue is that the script will issue the following command to kill the MQSAgent# process on the server that use to host the DTC/MQ Resource group. objShell.Run "cmd /k kill -f " & objProcess.ProcessId & "& exit The problem with this is that the kill command is not available on Windows 2003 and I suspect Windows 2008 as well. This command has been replaced with Taskkill. I opted to go a different route since the script they have provided is based upon WMI and VBscript I have replaced this line with objProcess.Terminate()
   

My testing to this point have been very successful. I have not had to manually intevene whatsoever and have not lost any messages when failing resources over or rebooting servers. The only delay that you will see is if you have messages inflight while you bounce resources is that if the Queue is still down that BizTalk will use the Send Port's configuration to issue retries. This is very standard BizTalk behaviour and is performing as expected.

So what does this script do? It essentially looks to see if the DTC service is running on the node that this script is running on. If the DTC service is running, the script will exit since we still need the MQSAgent to run on this server with DTC. If DTC is not running then it will look for a process called 'DLLHOST.EXE' that has a Process ID (PID) of the MQSAgent. If it finds an instance of this process it will terminate it so that this process does not lock up BizTalk from sending or receiving messages with MQ Series.

This sounds like an opportunity to cluster the MQSAgent application. However, the guidance from Microsoft is to not cluster it:

There is no requirement to cluster the MQSAgent (MQSAgent2) COM+ application that is used with the BizTalk Server MQSeries adapter. To provide high availability for this component, install the component on each cluster node. If the COM+ application stops, the next call from the client will start it.

I don't know...seems pretty clunky to me. I will have to follow up further with support.

Adventures with the HTTP Adapter and Yahoo Finance API

I was asked to investigate what would be involved in connecting BizTalk to a Yahoo Finance "api" in order to retrieve stock quotes. This is not a mission critical application but they wanted to be able to consume this information. The client "COTS" application can consume a WSDL, but not a HTTP response that includes comma delmitted data. So we figured that we could expose this data via a web service. There are certainly many ways to expose this data and this is not the point of this post.

The point of the post is to discuss some of the pitfalls that I ran into when trying to connect to this Yahoo API using the BizTalk HTTP Adapter. At first, I thought the problem was rather trivial, I opened IE, pointed it at the Yahoo URL, included the Stock Ticker and the format that I was interested in. The browser returned a string of data that included the stock quote and the other relevant data.

http://download.finance.yahoo.com/d/quotes.csv?s=MSFT&d=t&f=sl1d1t1c1ohgvj1pp2wern


I then saved a copy of this data into a text file, ran it through the BizTalk Flat file schema wizard, created a Receive Pipeline based upon this schema and now had "typed" data for use inside of BizTalk.

At this point, I was a little unsure of what Yahoo was expecting when I made this Http Request. I created a "dummy" schema which only had a root node and figured that I would submit the request to Yahoo to see what was going to happen. Initially, I had a static send port where I hard coded the URL. The URL was very important since it contains the HTTP Request query parameters. I figured that once I get this working, then I will focus on making it dynamic so that the client application can drive which stock quote is returned.

However, this is when I started to run into issues. When I tried to run my application using this configuration I was prompted with the following response from YAHOO: Missing Symbols List.

Based upon this error, I figured that something was up with the query parameters. Yahoo is expecting something along the lines of ?s=MSFT&d=t&f=sl1d1t1c1ohgvj1pp2wern to be past as part of the HTTP Request. After performing some BING searches someone suggested using a dynamic send port to pass these query parameters in. That didn't help either.

I then decided to open up Fiddler to see what was being passed as a successful request. Fiddler is a tool that can be used to inspect HTTP Requests and Responses.
When you use the Request Builder feature in Fiddler, it will default the HTTP request mode to "GET". It makes sense, but I then thought what if I switch this to POST? Look for the red line towards the bottom of the next image: "Missing Symbols List".

At this point, I am starting to understand the problem a little better. After another Bing search, I found the following document that indicates: "The HTTP send adapter gets messages from BizTalk Server and sends them to a destination URL on an HTTP POST request". Using Fiddler, I was able to determine that using a GET request worked without issue. Now knowing that the BizTalk HTTP Adapter is going to use a POST request, I figured that I needed to be able to get Fiddler to work with a POST request and then get BizTalk to use this same approach when posting data to Yahoo.

I am not going to get into the differences between POST and GET here as it has been done so many times before, but here is a good summary of the differences.

Since the query string is essentially being ignored anyways, I removed it from the URL Address text box. I then copied the query parameters into the "Request Body" text box without the '?'

Success! The next challenge is to get BizTalk to pass this data through the HTTP adapter as a POST request.

I found an old forum post by CranCran77 that discussed sending a message of type RawString to a different website that was also looking for HTTP Get. I have used the RawString class before when sending emails via BizTalk, so I was able to add this class to my project quickly.

In the image above, I have highlighted the "Construct Yahoo Request" Message Assignment shape. Below, I have the details of what is inside this message construct shape. Here I am assigning values to my message body that is of type "RawString". This RawString class has been added to a .Net Helper Assembly.

After the message is sent, I can look in the tracking database and can see that this data was transmitted as part of the message body.

Since the parameters that Yahoo requires are being sent as part of the message body, we may use a static Send Port and do not have to provide a query string.

With the emergence of SOAP and now WCF, the use of the HTTP adapter is limited. But as you can see there are still some "services" that exist on the web that rely upon HTTP. Unfortunately there is not a lot of good documentation on the HTTP adapter so hopefully this post fills in some of the gaps.