Another great post by Richard Seroter regarding BAM and the "OnlineWindowTimeLength" field. This field is used to determine how long data that resides in the BAMPrimaryImport views remains "live" before it is archived.
http://seroter.wordpress.com/2007/06/22/biztalk-bam-data-archiving-explained/
Monday, June 25, 2007
Sunday, June 24, 2007
/n software - FTP Adapter
I am currently involved in a project where we have a requirement to receive/send flat files from/to an AIX (UNIX) server using FTP.
The project uses BizTalk 2006, so a person may be asking why would you go out and purchase a third party adapter when BizTalk 2006 includes a Microsoft FTP Adapter in the product? I am not going to go out of my way to deliberately bring the Microsoft Adapter down, but we did have a few problems that were exposed in our performance tests that would prevent us from going live and having a lot of confidence in that adapter. One of the problems involved using a Temporary receive folder under heavy load and the inability to use this temporary receive folder in ASCII mode.
I had heard about the /n software FTP adapter earlier when someone asked me whether we could use Secure FTP or FTPS to when communicating with some external partners for a different project.
We were able to download a trial version of the adapter and were happy to find out that we could not duplicate the same problem that occurred when using the Microsoft Adapter and we could use ASCII mode when using a temporary receive folder while retrieving files from Unix.
For those that are not aware, Unix systems and Windows systems have different ways of indicating a Carriage Return/Line Feed. In Unix, this is represented by just a carriage return, also represented in Hex as 0D (zeroD) and in Windows as two characters (0D 0A) (zeroD zeroA). When using an FTP client, such as an adapter, the client will take care of the "translation" of these characters depending upon the target environment. When using Binary, the carriage returns are preserved "as is". So if you developed a schema with Child delimiter characters based upon the windows format and receive a document from Unix using Binary mode, you will end up with a parsing error. BizTalk will try to break up lines based upon the windows format, yet the message that was sent to BizTalk is in the Unix format. To get around this you need to manage multiple versions of Schemas if you need to receive a file from a Unix Box and send to a Windows box. This is a bit of hassle. Suffice to say, we were very happy that the /n software adapter supported temporary receive folders in ASCII mode.
As part of our project, we need to determine where to send certain files at runtime. Some destinations will be Unix and some will be windows. So sometimes we will be sending files out using FTP and sometimes using the File Adapter. Since we are determining this all at runtime, we try to use dynamic send ports and set the Microsoft.XLANGs.BaseTypes.Address property of the Dynamic sent port like below:
SendDynamicFlatFile(Microsoft.XLANGs.BaseTypes.Address)= completeDestinationURI;
We were a little surprised that when we tried to use this type of method with the /n software adapter that it would not work. What we were seeing was an error in the event viewer that essentially said something like "Unable to send file to server x, using this username ' ' ". What seemed to be happening is that the username context property for the adapter was not being set. We were able to check the context properties and the username and password was being set.
After exchanging a few emails with /n software support, we found that the adapter does not support using the Microsoft.XLANGs.BaseTypes.Address property via runtime code. So technically, the adapter does not support dynamic send ports. So I am going to get into some further details about how you can use the /n software adapter in a dynamic fashion by using a static send port.
Like most adapters, the /n software adapter exposes adapter context properties. You can explicitly set these properties at runtime in a message construction shape. When these properties are set via code, they are respected by the send port. In order to have these settings visible in a Message assignment window, you need to include a reference to nsoftware.BizTalk.FTPAdapter.
Once you have deployed your BizTalk project, you will now need to configure a physical static port in the BizTalk Admin Console. The minimal properties that need to be set in the send port are server, username and password. These settings can be completely irrelevant as they will not be used if you "override" them in your code. To verify this behaviour, I set my username to a non-existant username and provided a server name that also does not exist. When I ran my test, my FTP connection was established with the properties that I set in my orchestration in my message assignment shape. UPDATE: Please see this post (http://kentweare.blogspot.com/2007/08/n-software-ftp-adapter-update.html) as /n software has updated their software to support using these context properties as macros inside of BizTalk Admin.
A downside to all of this is if you have a message that failed during the send process, that it will use the URI of the static port and not the URI based upon the properties you used in your code. In a "true" dynamic send port you get to see the actual URI.
When comparing performance between the two adapters we noticed that the receive times where fairly comparable when receiving smaller and larger files. However, on the send side we noticed that the /n software adapter was significantly faster than the Microsoft adapter. We do have some requirements for sending some larger files (300+ mb). When sending a 380 mb file using the /n software adapter we found times in the 3 minute neighbourhood. When sending the same file, in the same scenario, we found the Microsoft Adapter was in the 25 minute neighbourhood. To ensure that we were not masking the problem, we used Send port subscriptions to ensure that we could isolate the problematic areas and stay out of the orchestrations. In order to determine these times, we used HAT and the pipeline begin/end times to record our durations.
It should be noted, that the way /n Software and Microsoft deal with temporary folders/filenames on the send side is a little different. Microsoft uses a temporary send folder where the file is written to this folder first. While the file is being written, a guid is used as its filename. Once the file has been completely written to the temp folder, it is then moved to the destination folder.
The /n software adapter, uses a temporary file extension instead of a temp folder. So as this file is being written, it will use the temporary extension that you provide it with. In our case, we were using ".tmp". When the complete file is written, the .tmp extension is dropped. So I am not sure if using this type of approach is the reason why the process is so much quicker. The delay for the Microsoft adapter is definitely not related to moving the file from the temp folder to the final folder. Using a Telnet client, or FTP Client, you are able to see the file being slowly written to the destination disk.
So all in all I think it depends on your circumstances as to which adapter you choose. The situations that I have just explained made the business case for us to use the /n Software adapter. The pricing for the /n Software Adapters is fairly reasonable. You cannot buy an adapter individually, as they come as a bundle including some additional adapters such as AS2, SFTP, Credit Card, RSS, SSH, Email, SMPP, SNPP, XMPP and S3.
Disclaimer:
I am not affiliated in any way with /n Software or any of its subsidiaries. I stand to gain nothing financially by posting this blog. I just thought that there may be some people who are in a similar situation who may enjoy some insight into this third party adapter.
I also do not have a hate on for Microsoft and its products. However, I am a firm believer in using the best tool for the job, in our situation the /n software adapter was the best option.
The project uses BizTalk 2006, so a person may be asking why would you go out and purchase a third party adapter when BizTalk 2006 includes a Microsoft FTP Adapter in the product? I am not going to go out of my way to deliberately bring the Microsoft Adapter down, but we did have a few problems that were exposed in our performance tests that would prevent us from going live and having a lot of confidence in that adapter. One of the problems involved using a Temporary receive folder under heavy load and the inability to use this temporary receive folder in ASCII mode.
I had heard about the /n software FTP adapter earlier when someone asked me whether we could use Secure FTP or FTPS to when communicating with some external partners for a different project.
We were able to download a trial version of the adapter and were happy to find out that we could not duplicate the same problem that occurred when using the Microsoft Adapter and we could use ASCII mode when using a temporary receive folder while retrieving files from Unix.
For those that are not aware, Unix systems and Windows systems have different ways of indicating a Carriage Return/Line Feed. In Unix, this is represented by just a carriage return, also represented in Hex as 0D (zeroD) and in Windows as two characters (0D 0A) (zeroD zeroA). When using an FTP client, such as an adapter, the client will take care of the "translation" of these characters depending upon the target environment. When using Binary, the carriage returns are preserved "as is". So if you developed a schema with Child delimiter characters based upon the windows format and receive a document from Unix using Binary mode, you will end up with a parsing error. BizTalk will try to break up lines based upon the windows format, yet the message that was sent to BizTalk is in the Unix format. To get around this you need to manage multiple versions of Schemas if you need to receive a file from a Unix Box and send to a Windows box. This is a bit of hassle. Suffice to say, we were very happy that the /n software adapter supported temporary receive folders in ASCII mode.
As part of our project, we need to determine where to send certain files at runtime. Some destinations will be Unix and some will be windows. So sometimes we will be sending files out using FTP and sometimes using the File Adapter. Since we are determining this all at runtime, we try to use dynamic send ports and set the Microsoft.XLANGs.BaseTypes.Address property of the Dynamic sent port like below:
SendDynamicFlatFile(Microsoft.XLANGs.BaseTypes.Address)= completeDestinationURI;
We were a little surprised that when we tried to use this type of method with the /n software adapter that it would not work. What we were seeing was an error in the event viewer that essentially said something like "Unable to send file to server x, using this username ' ' ". What seemed to be happening is that the username context property for the adapter was not being set. We were able to check the context properties and the username and password was being set.
After exchanging a few emails with /n software support, we found that the adapter does not support using the Microsoft.XLANGs.BaseTypes.Address property via runtime code. So technically, the adapter does not support dynamic send ports. So I am going to get into some further details about how you can use the /n software adapter in a dynamic fashion by using a static send port.
Like most adapters, the /n software adapter exposes adapter context properties. You can explicitly set these properties at runtime in a message construction shape. When these properties are set via code, they are respected by the send port. In order to have these settings visible in a Message assignment window, you need to include a reference to nsoftware.BizTalk.FTPAdapter.
Once you have deployed your BizTalk project, you will now need to configure a physical static port in the BizTalk Admin Console. The minimal properties that need to be set in the send port are server, username and password. These settings can be completely irrelevant as they will not be used if you "override" them in your code. To verify this behaviour, I set my username to a non-existant username and provided a server name that also does not exist. When I ran my test, my FTP connection was established with the properties that I set in my orchestration in my message assignment shape. UPDATE: Please see this post (http://kentweare.blogspot.com/2007/08/n-software-ftp-adapter-update.html) as /n software has updated their software to support using these context properties as macros inside of BizTalk Admin.
A downside to all of this is if you have a message that failed during the send process, that it will use the URI of the static port and not the URI based upon the properties you used in your code. In a "true" dynamic send port you get to see the actual URI.
When comparing performance between the two adapters we noticed that the receive times where fairly comparable when receiving smaller and larger files. However, on the send side we noticed that the /n software adapter was significantly faster than the Microsoft adapter. We do have some requirements for sending some larger files (300+ mb). When sending a 380 mb file using the /n software adapter we found times in the 3 minute neighbourhood. When sending the same file, in the same scenario, we found the Microsoft Adapter was in the 25 minute neighbourhood. To ensure that we were not masking the problem, we used Send port subscriptions to ensure that we could isolate the problematic areas and stay out of the orchestrations. In order to determine these times, we used HAT and the pipeline begin/end times to record our durations.
It should be noted, that the way /n Software and Microsoft deal with temporary folders/filenames on the send side is a little different. Microsoft uses a temporary send folder where the file is written to this folder first. While the file is being written, a guid is used as its filename. Once the file has been completely written to the temp folder, it is then moved to the destination folder.
The /n software adapter, uses a temporary file extension instead of a temp folder. So as this file is being written, it will use the temporary extension that you provide it with. In our case, we were using ".tmp". When the complete file is written, the .tmp extension is dropped. So I am not sure if using this type of approach is the reason why the process is so much quicker. The delay for the Microsoft adapter is definitely not related to moving the file from the temp folder to the final folder. Using a Telnet client, or FTP Client, you are able to see the file being slowly written to the destination disk.
So all in all I think it depends on your circumstances as to which adapter you choose. The situations that I have just explained made the business case for us to use the /n Software adapter. The pricing for the /n Software Adapters is fairly reasonable. You cannot buy an adapter individually, as they come as a bundle including some additional adapters such as AS2, SFTP, Credit Card, RSS, SSH, Email, SMPP, SNPP, XMPP and S3.
Disclaimer:
I am not affiliated in any way with /n Software or any of its subsidiaries. I stand to gain nothing financially by posting this blog. I just thought that there may be some people who are in a similar situation who may enjoy some insight into this third party adapter.
I also do not have a hate on for Microsoft and its products. However, I am a firm believer in using the best tool for the job, in our situation the /n software adapter was the best option.
Subscribe to:
Posts (Atom)