Sunday, April 20, 2008

FtpWebRequest Class: Creating Remote Directories


So a question that you are probably asking yourself is, if BizTalk 2006 Rx has a FTP adapter that ships out of the box, why am I concerning myself with this .Net Class? The reason why I am concerning myself with this is that if you need to ensure that a remote folder exists, before you write the file, you can use this class to execute FTP commands against a remote system.

My Requirements

I need to connect to a Unix file system and deliver 3 files: a "work" file, a "sig" file and an "archive" file. So if you had a static file structure, this is not a difficult feat. My issue is that the archive file needs to be placed in the current day's directory. The directory structure that I need to deliver to is /arch/yyyy/mm/dd.

The way the class behaves is that I cannot just assume that the parent folder exists. If I try to determine if 2008/01 exists, I will get an exception, but I won't be able to determine which folder does not exist. So based upon this, I need to ensure that the correct Year folder exists, then month and then day.

Solution Options

  1. Write a Unix script that is included in a Crontab job that routinely checks for missing folders, when a missing folder is encountered, the script creates it.

  2. As part of the BizTalk process, for each file that is about to be written, use the FtpWebRequest Class to check to see if the folder(s) exist. If they do not, then create them as required.

  3. Much like Option #2, have BizTalk create the folder structures, but create a routine that runs once a day and creates folders out several days in advance.

  4. Manually create the folders and "assume" they exist.

Solution Chosen

I chose Option #3 for a few reasons:

  1. I come from a .Net background and am not all that anxious to get my hands dirty writing Unix scripts.

  2. While Option #2, would work it would create unnessary connections to the Unix System during every archive file write. It wouldn't kill the performance, but is not ideal.

  3. Option #3 runs once a day and I can schedule it so that it runs during off-peak load. I can use MOM to notify me of any exceptions that occur and use the SMTP adapter to send me a summary of the folders that were created successfully.

So what is the big fuss?

During the development of the solution, I ran into some interessing quirks about the FtpWebRequest Class. There is not a lot of documentation on it and there seems to be ample problems with it on the forums.

So to start with, familiarize your self with the MSDN documentation: http://msdn2.microsoft.com/en-us/library/system.net.ftpwebrequest.aspx

FtpWebRequest Class Behaviour

  1. If you try to execute a directory listing on an empty folder, a
    System.Net.WebException will be raised. For instance if a person is looking for all folders in a directory called /2008 and there are no child folders in this folder, an exception will be raised. I find this behaviour to be extremely odd and I think this creats a lot of confusion for developers.

  2. I have not found a way to 'defensively' code against scenario #1. The only way that I have been able to deal with this situation is to catch the exception and then create the new folder. I do realize that it is not best practice to use exception handlers as a way to deal with logic, but in this class I did not find a way around it.

  3. If child folders to exist in the directory being listed, a stream is returned which can easily be assigned to a string. You then need to search the string to see if the folder already exists. Also note that in this list, you will find Carriage Return/Line Feed characters which are represented as \r\n

  4. If your FTP Server has different file mounts and you need to navigate up the tree, you can use '%2E%2E' which is the equivalent of '..' So for instance when I log into my FTP server, I get logged into /home/'username'. The path to my archive directory is /'company'/'environment'/interfaces/'interface'/inbound/arch/ . So before I check to see if the archive folder exists, I want to go up two directories, before I navigate to it. '/home' is a sibling folder of '/company' The way this looks in code is:
    string ftpURI = "ftp://" + server + "/%2E%2E/%2E%2E" + path;

The Code

So my purpose for this code is to see if a target directory exists, if it does not exist, I want to create it. To simplify the code, I have removed determining if the Month and Day exist, but the concept remains the same. Ultimately, the return string is my entire archive path: /yyyy/MM/dd . If I encounter an exception that is unexpected ( something other than a System.Web.Exception) I throw an exception back to the caller.

Here is some sample code that can be called from .Net or BizTalk if you put it into a Serializable attributed class:


public string CreateNewUnixArchiveFolder(string path, string server, string username, string password, string currentYear, string currentMonth, string currentDay)

{

string tmpStr = "";

FtpWebResponse FTPResp;
FtpWebRequest request;
Stream ftpRespStream;
StreamReader sr;
string folderList;
string ftpURI = "ftp://" + server + "/%2E%2E/%2E%2E" + path;
tmpStr = currentYear + currentMonth + currentDay;

try
{
request = (FtpWebRequest)WebRequest.Create(ftpURI);
request.KeepAlive = false;
request.Method = WebRequestMethods.Ftp.ListDirectory;
request.Credentials = new NetworkCredential(username, password);
try
{
FTPResp = (FtpWebResponse)request.GetResponse();
ftpRespStream = FTPResp.GetResponseStream();
sr = new StreamReader(ftpRespStream, Encoding.Default);
folderList = sr.ReadToEnd();

//close the objects
ftpRespStream.Close();
request = null;

//save the list into a string
if (!folderList.Contains(currentYear.Replace("/", "") + "\r\n"))
{ // Parent folder is not empty so no exception thrown, but current Year does not exist
//Current Year does not exist in archive...need to create it
request = (FtpWebRequest)WebRequest.Create(ftpURI + currentYear);
request.KeepAlive = false;
request.Method = WebRequestMethods.Ftp.MakeDirectory;
request.Credentials = new NetworkCredential(username, password);

FTPResp = (FtpWebResponse)request.GetResponse();
ftpRespStream = FTPResp.GetResponseStream();

//close the objects
request = null;
ftpRespStream.Close();
}
}
catch (System.Net.WebException)
{
//Current Year does not exist in archive...need to create it
request = (FtpWebRequest)WebRequest.Create(ftpURI + currentYear);
request.KeepAlive = false;
request.Method = WebRequestMethods.Ftp.MakeDirectory;
request.Credentials = new NetworkCredential(username, password);
FTPResp = (FtpWebResponse)request.GetResponse();
ftpRespStream = FTPResp.GetResponseStream();

//close the objects
request = null;
ftpRespStream.Close();
}
}
catch (Exception ex)
{
throw ex;
}
return tmpStr;
}
 

2008 MVP Global Summit: Canadians @ MVP Summit 2008

This past week, April 14 th – 18th, I attended the 2008 MVP Global Summit. It was an opportunity for Microsoft MVPs, to come out to Seattle/Redmond to meet with the product teams to talk about the direction of their product(s). It also provided an opportunity to collaborate with other MVPs through Regional MVP Dinners, Product Team Dinners and in Open Space Sessions.

Led by our fearless leader, Sasha Krsmanovic, there was a strong contingent of Canadian MVPs in attendance. I believe the total number was 110 which was #2 in terms of country representation. At the 2007 Summit, a bit of a tradition was introduced that included Canadian MVPs wearing Red Canadian National Hockey Jerseys. This tradition continued in 2008 as all new MVPs, in attendance, were provided with a new jersey.

A new concept this year included a Hockey Puck hunt. Each Canadian MVP was given, at least, 3 different hockey pucks to distribute to other attendees. Some Canadian MVPs had the special “MVP” puck. For the Non-Canadian attendees, the goal was to try and collect all 4 pucks. Once you had all 4 pucks, you could then turn them into Sasha for your very own Team Canada jersey. This was an excellent idea and provided for another opportunity meet new people from other countries. At the end of the summit, all Canadian MVPs were given a set of the 4 pucks to take home.



Some other attendees loved these ideas while some others felt we were suffering from “Look at me syndrome”. All in all, I think the ideas were great and the intent was not to rub people the wrong way. The intent was purely to provide Canadian MVPs with an opportunity to meet new people. I met a lot of people that I may not have met had it not been for the Jersey and/or pucks. So I would say ...Mission Accomplished.

So all in all, it was a great event. Here are some of my highlights:

  • Meeting with other Connected Systems Division (CSD) MVPs and the Connected Systems Division Product teams to discuss upcoming plans.

  • Hanging out with other CSD MVPs at the dinners and outside the sessions. This is a very international bunch, so it was interesting to get different perspective on how other regions around the world use BizTalk and .Net. I have met a couple of the MVPs at other events, but it was great to meet some of the people, whose blog or forum posts I read, face to face.
  • The Keynote presentations by Ray Ozzie and Steve Ballmer. I enjoyed both executive’s passion for the technology and Steve has the energy that you usually only see on NFL.


(Steve Ballmer sporting a Team Canada Hockey Jersey)