FtpWebRequest Class: Creating Remote Directories

So a question that you are probably asking yourself is, if BizTalk 2006 Rx has a FTP adapter that ships out of the box, why am I concerning myself with this .Net Class? The reason why I am concerning myself with this is that if you need to ensure that a remote folder exists, before you write the file, you can use this class to execute FTP commands against a remote system.

My Requirements

I need to connect to a Unix file system and deliver 3 files: a "work" file, a "sig" file and an "archive" file. So if you had a static file structure, this is not a difficult feat. My issue is that the archive file needs to be placed in the current day's directory. The directory structure that I need to deliver to is /arch/yyyy/mm/dd.

The way the class behaves is that I cannot just assume that the parent folder exists. If I try to determine if 2008/01 exists, I will get an exception, but I won't be able to determine which folder does not exist. So based upon this, I need to ensure that the correct Year folder exists, then month and then day.

Solution Options

1. Write a Unix script that is included in a Crontab job that routinely checks for missing folders, when a missing folder is encountered, the script creates it.


2. As part of the BizTalk process, for each file that is about to be written, use the FtpWebRequest Class to check to see if the folder(s) exist. If they do not, then create them as required.


3. Much like Option #2, have BizTalk create the folder structures, but create a routine that runs once a day and creates folders out several days in advance.


4. Manually create the folders and "assume" they exist.

Solution Chosen

I chose Option #3 for a few reasons:

1. I come from a .Net background and am not all that anxious to get my hands dirty writing Unix scripts.


2. While Option #2, would work it would create unnessary connections to the Unix System during every archive file write. It wouldn't kill the performance, but is not ideal.


3. Option #3 runs once a day and I can schedule it so that it runs during off-peak load. I can use MOM to notify me of any exceptions that occur and use the SMTP adapter to send me a summary of the folders that were created successfully.
   

So what is the big fuss?

During the development of the solution, I ran into some interessing quirks about the FtpWebRequest Class. There is not a lot of documentation on it and there seems to be ample problems with it on the forums.

So to start with, familiarize your self with the MSDN documentation: http://msdn2.microsoft.com/en-us/library/system.net.ftpwebrequest.aspx

FtpWebRequest Class Behaviour

1. If you try to execute a directory listing on an empty folder, a
   System.Net.WebException will be raised. For instance if a person is looking for all folders in a directory called /2008 and there are no child folders in this folder, an exception will be raised. I find this behaviour to be extremely odd and I think this creats a lot of confusion for developers.


2. I have not found a way to 'defensively' code against scenario #1. The only way that I have been able to deal with this situation is to catch the exception and then create the new folder. I do realize that it is not best practice to use exception handlers as a way to deal with logic, but in this class I did not find a way around it.


3. If child folders to exist in the directory being listed, a stream is returned which can easily be assigned to a string. You then need to search the string to see if the folder already exists. Also note that in this list, you will find Carriage Return/Line Feed characters which are represented as \r\n


4. If your FTP Server has different file mounts and you need to navigate up the tree, you can use '%2E%2E' which is the equivalent of '..' So for instance when I log into my FTP server, I get logged into /home/'username'. The path to my archive directory is /'company'/'environment'/interfaces/'interface'/inbound/arch/ . So before I check to see if the archive folder exists, I want to go up two directories, before I navigate to it. '/home' is a sibling folder of '/company' The way this looks in code is:
   string ftpURI = "ftp://" + server + "/%2E%2E/%2E%2E" + path;

The Code

So my purpose for this code is to see if a target directory exists, if it does not exist, I want to create it. To simplify the code, I have removed determining if the Month and Day exist, but the concept remains the same. Ultimately, the return string is my entire archive path: /yyyy/MM/dd . If I encounter an exception that is unexpected ( something other than a System.Web.Exception) I throw an exception back to the caller.

Here is some sample code that can be called from .Net or BizTalk if you put it into a Serializable attributed class:

public string CreateNewUnixArchiveFolder(string path, string server, string username, string password, string currentYear, string currentMonth, string currentDay)

        {

            string tmpStr = "";

            FtpWebResponse FTPResp;
            FtpWebRequest request;
            Stream ftpRespStream;
            StreamReader sr;
            string folderList;
            string ftpURI = "ftp://" + server + "/%2E%2E/%2E%2E" + path;
            tmpStr = currentYear + currentMonth + currentDay;

            try
            {
                request = (FtpWebRequest)WebRequest.Create(ftpURI);
                request.KeepAlive = false;
                request.Method = WebRequestMethods.Ftp.ListDirectory;
                request.Credentials = new NetworkCredential(username, password);
                try
                {
                    FTPResp = (FtpWebResponse)request.GetResponse();
                    ftpRespStream = FTPResp.GetResponseStream();
                    sr = new StreamReader(ftpRespStream, Encoding.Default);
                    folderList = sr.ReadToEnd();
                    //close the objects
                    ftpRespStream.Close();
                    request = null;

                    //save the list into a string
                    if (!folderList.Contains(currentYear.Replace("/", "") + "\r\n"))
                    {   // Parent folder is not empty so no exception thrown, but current Year does not exist
                        //Current Year does not exist in archive...need to create it
                        request = (FtpWebRequest)WebRequest.Create(ftpURI + currentYear);
                        request.KeepAlive = false;
                        request.Method = WebRequestMethods.Ftp.MakeDirectory;
                        request.Credentials = new NetworkCredential(username, password);

                        FTPResp = (FtpWebResponse)request.GetResponse();
                        ftpRespStream = FTPResp.GetResponseStream();

                        //close the objects
                        request = null;
                        ftpRespStream.Close();
                    }
                }
                catch (System.Net.WebException)
                {
                    //Current Year does not exist in archive...need to create it
                    request = (FtpWebRequest)WebRequest.Create(ftpURI + currentYear);
                    request.KeepAlive = false;
                    request.Method = WebRequestMethods.Ftp.MakeDirectory;
                    request.Credentials = new NetworkCredential(username, password);
                    FTPResp = (FtpWebResponse)request.GetResponse();
                    ftpRespStream = FTPResp.GetResponseStream();

                    //close the objects
                    request = null;
                    ftpRespStream.Close();
                }
            }
            catch (Exception ex)
            {
                throw ex;
            }
            return tmpStr;
        }

 FtpWebRequest

2008 MVP Global Summit: Canadians @ MVP Summit 2008

This past week, April 14 th – 18th, I attended the 2008 MVP Global Summit. It was an opportunity for Microsoft MVPs, to come out to Seattle/Redmond to meet with the product teams to talk about the direction of their product(s). It also provided an opportunity to collaborate with other MVPs through Regional MVP Dinners, Product Team Dinners and in Open Space Sessions.

Led by our fearless leader, Sasha Krsmanovic, there was a strong contingent of Canadian MVPs in attendance. I believe the total number was 110 which was #2 in terms of country representation. At the 2007 Summit, a bit of a tradition was introduced that included Canadian MVPs wearing Red Canadian National Hockey Jerseys. This tradition continued in 2008 as all new MVPs, in attendance, were provided with a new jersey.

A new concept this year included a Hockey Puck hunt. Each Canadian MVP was given, at least, 3 different hockey pucks to distribute to other attendees. Some Canadian MVPs had the special “MVP” puck. For the Non-Canadian attendees, the goal was to try and collect all 4 pucks. Once you had all 4 pucks, you could then turn them into Sasha for your very own Team Canada jersey. This was an excellent idea and provided for another opportunity meet new people from other countries. At the end of the summit, all Canadian MVPs were given a set of the 4 pucks to take home.

Some other attendees loved these ideas while some others felt we were suffering from “Look at me syndrome”. All in all, I think the ideas were great and the intent was not to rub people the wrong way. The intent was purely to provide Canadian MVPs with an opportunity to meet new people. I met a lot of people that I may not have met had it not been for the Jersey and/or pucks. So I would say ...Mission Accomplished.

So all in all, it was a great event. Here are some of my highlights:

• Meeting with other Connected Systems Division (CSD) MVPs and the Connected Systems Division Product teams to discuss upcoming plans.
  


• Hanging out with other CSD MVPs at the dinners and outside the sessions. This is a very international bunch, so it was interesting to get different perspective on how other regions around the world use BizTalk and .Net. I have met a couple of the MVPs at other events, but it was great to meet some of the people, whose blog or forum posts I read, face to face.

• The Keynote presentations by Ray Ozzie and Steve Ballmer. I enjoyed both executive’s passion for the technology and Steve has the energy that you usually only see on NFL.

(Steve Ballmer sporting a Team Canada Hockey Jersey)

Canadians @ MVP Summit 2008

Processing Inter-dependent files using a Non-Uniform Sequential Convoy

The Problem

A re-occurring topic that I have recently come across on some of the BizTalk forums(here and here) is the ability to process interdependent files. More specifically we want the ability to process a particular file after receiving a "trigger" or "signal" file.

The use of trigger and signal files is very common in SAP and mainframe systems. A common pattern in the SAP world is to write "work" files to a "work" folder and a "signal" file to a "sig" folder. For SAP outbound files, the SAP system will write data files to the work folder, once the file has been completely written, a signal file will be written to the sig folder. This sig file gives the Middleware, or downstream system, an indication that this file is safe for processing. This is important as some files that get written from these types of systems are very large or may be written to over a period of time via batch jobs.

BizTalk supports messaging patterns that provide the ability to wait for messages to arrive before completing a business process(or orchestration). These mechanisms tend to use correlation and the use of convoys. Stephen W Thomas has written a whitepaper that dives into these topics further.

The messaging pattern that I have chosen to aid in this solving this challenge is the Non-Uniform Sequential Convoy. This pattern's mandate is to process 2 or more messages in a known order.

The challenge with this pattern, for our scenario, is that our work file is written prior to the sig file, but in terms of BizTalk processing messages we want BizTalk to only process the work file once the sig file has been written.

In order to solve this problem we are going to leverage a .Net Helper class to do some of the lifting.

Solution

I have written a proof of concept(POC) app to demonstrate the pattern. The solution is pretty light and easy to implement. It contains 2 message schemas(1 for the signal file, 1 for the work file), a property schema, an orchestration and a .Net Helper class. Note, that I have left what you do with both files once you get them into the same instance of the orchestration out of scope. At this point your requirements will determine what you need to do with both files.

The first artifact that will dive into is the Signal file. The file itself does not need to contain a lot of data. For my sample I have two elements, a timestamp and a WorkFileName. In order to use my convoy, I need to create a correlation type and correlation set. A correlation type requires a promoted property(or BizTalk System property) in order to "link" multiple messages to one running instance of an orchestration. For more information on Correlation, please see the following document.

The second artifact represents data that could be generated by an upstream system such as SAP or a Mainframe. This document is entirely fictitious so don't look too much into it. Also note that I have a promoted property called FileName that is used in my Correlation Type.

So for this example I am using FileName but you can correlate based upon any data, as long as your signal file and work file promote the same data values.

Below is a snapshot of what our orchestration looks like.

A few things to note is that Non-Uniform Sequential convoys to require that both receive shapes connect to the same receive port. This logical receive port also needs to be marked for Ordered Delivery.




For the initial receive shape we need to set a few properties. Since this is the first Receive shape we need to set the Activate property to True in order to instantiate the orchestration.

The other property that we need to populate is the Initializing Correlation Sets. In order for BizTalk to "wait" for the work file to be picked up by the same instance of the orchestration that consumed the signal file we need to initialize a correlation set. (Keep reading for more info on how to create the Correlation Set).


Prior to creating a Correlation Set, you need to create a Correlation Type. You can do this from the Orchestration View tab.

We want to create the Correlation Type based upon the element/attribute that we promoted in each of the two schemas.
We then need to create a Correlation Set, which is instantiated by the initial Receive Shape, that is based upon the Correlation Type that was just created. Correlation Sets are also configured in the Orchestration View tab.

In the Rename Work File Expression shape we are going to call a .Net Helper class that will aid in renaming work file in the source folder. This step is a critical step in the process. Since the work file is completely written before the sig file is, we cannot use the original file extension in the Receive Location file mask. Otherwise, this would prompt BizTalk to consume the work file prior to us wanting it to be consumed. By appending a temporary extension, like .BIZ, to the end of the work file name, we can be assured that BizTalk will pick the file up when we want it to. So in the Receive Location we will use a *.BIZ extension instead of a *.XML.

Note that I have hard coded the path of the source location. Since this is a POC, this is ok but this would not be a suitable solution for a production environment

In order for BizTalk to "wait" for the work file to be picked up, we need to set the Following Correlation Sets property with the same Correlation Set that we initialized in the first Receive shape. Since the rename operation occurs the step before, the "wait" time will be extremely small. The main point here is that we want to control when BizTalk picks up the work file. It is essentially the rename operation and the second receive shape/location that controls this.

Now that you have consumed both the signal file and work file, in order, you can finish up any processing that is required by your business requirements. Since this is just a POC, I output the work file to a folder.


Testing
In order to simulate how an upstream system would write the files, I drop a file into the work folder with the original extension. I then drop a sig file into the sig folder. The sig file will get picked up and BizTalk, via the .Net Helper, will rename the work file. The receive location, for work files, will then pick up the work file and the orchestration will finish processing both files.


Conclusion
Through the use of a Messaging Pattern and with the help of a .Net Helper class we can process a set of files in a known order.

TopXML Oracle Adapter Dynamic "like" Send Port

A colleague and I recently ran into a situation where we needed to determine the Oracle Connection string at runtime. This may sound like an odd requirement, but the 3rd party application that populates this database has been configured for redundancy in an a-typical fashion. It has not been set up in a fashion like a SQL Cluster...that would be too easy :-). So instead of letting an infrastructure component(like Windows Clustering) determine which instance is the "Primary", their software performs this function.

I have set up a POC that will walk you through this configuration.

The first thing that you are going to want to do is add a reference to your project. The DLL that you want to include is "BTSUtils.Adapters.Databases.Properties.dll". You can find this file in your \BizTalk Utilities\Database Adapter\Bin.

This will allow you to update the Connectionstring context property.

Below is an image of my orchestration. The orchestration doesn't do anything all that spectacular. The majority of the orchestration is for demonstration purposes.

Basically, a file is passed in that has data which will be used as part of the Oracle Request. So a simple map will take the value from the incoming document and map it to the Oracle Request.

Once, we have the Oracle Request document, we need to update the context property as part of the Message Assignment Shape. I have updated the
BTSUtils.Adapters.Databases.Properties.Transmit.Connectionstring and assigned a new value.

So obviously hard coding the connection string in your orchestration is not a good idea, but remember this is just some POC type code. In our requirements, we need to be able to connect to one of two data sources, but have to determine this at run time. Some options for storing these connections strings may be the BTSNTSvc.exe.config(clear text warning!) or in the SSODB(secure) and extracting the values via the SSO API.

Once this property is set, I submit the Oracle Request message and wait for a Response. The Expression shape in my orchestration, is simply a trace statement. I am interested in knowing how may nodes are returned as a result of my query.

After this log point, I simply drop the results of the query to a folder.

So if I modify the connection string inside of the Orchestration, then what does the Send Port configuration look like?

Interestingly enough, you do not have to populate the "Connection String" property with anything meaningful what so ever. You do need to provide it with a value in order to successfully complete the configuration though. I have decided to populate this property with the associated Macro. This way an Ops guy can look at it and figure out that the value is populated elsewhere.

Microsoft MVP Awarded: Windows Server System - BizTalk Server

I was recently informed that I have received MVP status from Microsoft. It is truly an honor to be recognized by Microsoft for this award. I look forward to the MVP experience and meeting other MVPs at Microsoft events such as the MVP Global Summit.

If you would like to see my MVP profile, you may see it here.

WMI Scripting for BizTalk Operations Tasks

While reading a BizTalk Forum on MSDN, I came across a post asking for guidance on detecting when a Receive Location is disabled that did not include MOM. I knew that a lot of information is retrievable via WMI, but I never investigated it much because we use MOM at the organization that I am with.

So I started to dig into WMI some more and came across this post that describes how you can added the managed WMI classes to your .Net application(Console/Winform/Service etc). http://geekswithblogs.net/gwiele/archive/2005/03/16/26469.aspx

I looked for a class that could help out with a disabled receive location and found this class: MSBTS_ReceiveLocation. When I selected "Generate Managed Class" a file was added to my solution called: ROOT.MicrosoftBizTalkServer.MSBTS_ReceiveLocation.cs
With this class added to my solution, detecting whether or not the receive location was enabled was a breeze:

WMITest.ROOT.MICROSOFTBIZTALKSERVER.ReceiveLocation rl =new WMITest.ROOT.MICROSOFTBIZTALKSERVER.ReceiveLocation("yourmanagementdb", "yourservername", "receivelocationname","receiveportname");

if (rl.IsDisabled)
{
rl.Enable();
}

Another Class that I can see some value in using is the MSBTS_HostInstance class. This took me a few minutes to figure out. In order to create an object instance of one of Host Intances run the following code:

WMITest.ROOT.MICROSOFTBIZTALKSERVER.HostInstance hi =new WMITest.ROOT.MICROSOFTBIZTALKSERVER.HostInstance("yourmanagementdb", "servername", "Microsoft BizTalk Server Hostinstancename servername");

if (hi.ServiceState == WMITest.ROOT.MICROSOFTBIZTALKSERVER.HostInstance.ServiceStateValues.Stopped)
{
hi.Start();
}

The last parameter when instantiating the object threw me for a loop. In order to figure exactly what the WMI call was expecting, I decided to check out the BizTalkMgmtDb to figure out what the "internal" name was. I found this value in the adm_HostInstance table in the Name column.

Overall it was a good experience playing with some of this WMI. As time permits I plan on doing some additional investigation.

BizTalk Delivery Notification

I have recently had some requirements that required me to use the BizTalk Delivery Notification feature of Send Ports. In case you didn't know, Delivery Notification "Gets or sets the field to request that the adapter will send delivery notification back, whether transmission is successful. " source

Basically, what I am trying to do is ensure that my message is sent to it's destination prior to the rest of the orchestration executing. If the message is not sent successfully, I want the orchestration to stop processing so that an "Ops guy" can take a look into the situation without losing the message.

My Requirements

I need to deliver a file to a Unix system where it will be consumed by our ERP system. Since some of the files may be large, it may take seconds or even minutes for the file to be completely written. Our ERP system essentially needs to know when it is "safe" to consume the the file. Once the data file, from here on in I will refer to it as the "work" file as it needs to be delivered to a "work" folder, is written a "signal" file is to be written to the "sig" folder. This is the ERP's trigger to pick up the work file. When writing the sig file, we use the same name as the work file, except we append ".sig" onto the end. This way the ERP knows which file to pick up in the work folder.

So what I need to do is ensure that the "work" file has been successfully written before the "sig" file. Otherwise our ERP may go looking for a "work" file that does not exist.

Solution

I have created a POC to demonstrate how to utilize Delivery Notification. For this sample I am using local file drops, but I have tested this with an FTP adapter as well.


So here is my orchestration. I have a standard receive shape followed by a Scope shape that has Transaction = None, Synchronized=True. To find out more info on Scope shapes click here

Inside of this scope, I have a log point and Send Shape that is connected to a Logical Send port. Within the Send Port, I have specified the "Delivery Notification" = Transmitted.

I have added an Exception handler to the scope shape that is configured to catch Microsoft.XLANGs.BaseTypes.DeliveryFailureException exceptions. When you enable Delivery Notification the orchestration will wait for an ACK/NACK to be returned when the message is sent. If an ACK is returned, the orchestration will complete the scope shape and continue processing. If a NACK is sent back from the engine then the orchestration will raise a Microsoft.XLANGs.BaseTypes.DeliveryFailureException which is why we need to catch this type of exception.

I have added a Suspend Orchestration shape to ensure that I can resume this orchestration instance in the event of a failure.

Outside the scope shape, I have another log point and I construct my sig file. There is not much to the sig file. It basically contains a timestamp. I do not have Delivery Notification enable for this logical send port as I will just rely upon the messaging engine's "default" capabilities for writing this file.

Testing

Under normal circumstances, the sample runs as expected. A file is picked up from the receive location where BizTalk tries to write the work file to the work folder and the sig file to the sig folder.

In the event that the work folder does not exist or the work file already exists, the sig file will not be written. In this test run, I have left a copy of the same file in the work folder. When the orchestration executes, I find 2 suspended instances in my BizTalk Admin Console. Note that for this test I have my Send port's Retry Count and Retry Interval set to 0 just to speed up the testing process. You can alter these values and the behaviour remains the same; the work file will be retried as many times as configured before it gets to this state.

At this point, in my production environment, I would receive a MOM Alert as BizTalk has logged an error in the Event Viewer. This would allow for some investigation into the problem. In my sample scenario I just remove the original file from this folder and am free to resume the Suspended Message Instance. Once resumed, the work file is written as it should be. In order to get the sig file written, I must now resume the Orchestration instance. Upon resuming this instance, the sig file is written to disk.

The order of these two steps is critical. If you resume the orchestration instance prior to the message instance, the sig file would be written prior to the work file. This is the scenario that we are trying to avoid by using Delivery Notification so it is important for operations staff to understand this concept.

Also be cautious about the potential performance impacts of enabling Delivery Notification. For more information related to performance, please check out Kevin Smith's blog