BizTalk Adapter Pack 2.0 – SAP Adapter IDoc Schema Versioning Part 2

In my first post, I discussed how the BizTalk Adapter pack Consume Adapter Service wizard tightly couples the IDoc schema version with the version of SAP that you used to generate the schema with.  The goal of this post is to discover how you can avoid this tight coupling and hopefully survive an SAP upgrade.  I know the organization that I work for would not be pleased with having to update all of our IDoc schemas when we do our next SAP Upgrade just because the version of SAP has been incremented.

When the BizTalk Adapter Pack came out I was pretty excited that I could throw away my Flat File schemas and pipelines.  While true, it comes at a cost.  When using the XML Receive pipeline with the new Adapter, the Adapter will use data in the SAP Control record to determine what version of the message you retrieving from SAP and will assign a namespace to your message, that contains SAP version information, before depositing it in the message box. If have not changed your SAP version since generating your IDocs you are in business.  If you have upgraded your SAP system or have generated the wrong schemas for your SAP version then you will be in a lot of pain. 

To get around this tight coupling issue, you can revert back to the old style of IDoc processing which involves flat files and pipelines.  Now there is nothing really wrong with this model, but it does require a few extra steps and is not as “clean” a solution as using the XML Receive pipeline.

I will now take you through how you generate a flat file schema that can be used to process different versions of the same IDoc.  When I say different schemas this means that the version, and therefore namespace, are different but structurally the documents are the same.  Note that I will not go through all steps required to generate an IDoc schema, only those that are important to generate a Flat File schema.  To see a more comprehensive walk through of generating an IDoc schema, please see my Webcast.

 

• When using Consume Adapter Service wizard, ensure that “GenerateFlatFileCompatible” is set to true.

When you set the GenerateFlatFileCompatible property to true, you will now see the flat file specific data within the Schema specification.

Without setting GenerateFlatFileCompatible ,  the schema will not contain this flat file information.

• You also want to indicate the “ReceiveIDocFormat” will be a String.  You will find out why this is important later on in this post.

• Once the schemas have been imported into Visual Studio, enable “Flat File Extensions” to the “core schema”.  When using the BizTalk Adapter Pack Consume Adapter Service wizard you will generate more than 1 schema unlike the old .Net Connector version where you would always only generate 1 schema.  By “core schema” I mean the one that contains the “meat” of your schema, not one that contains the shell(all of the imported schemas) or any of the base type schemas.

• You will now need to create a Receive Pipeline and add a Flat File disassembler

• Add your “shell” schema to the document schema property

• Build and Deploy your application
• When configuring your SAP Receive location you will want to specify your Receive Pipeline as opposed to XML Receive

• On the Binding tab, set the “ReceiveIdocFormat” to String

• On the Messages tab, provide an XPath expression that will extract the flat file data so it can be disassembled by the Receive Pipeline.  The XPath expression that I used is:

  /*[local-name()='ReceiveIdoc']/*[local-name()='idocData']

  The adapter is essentially wrapping this flat file in XML tags so that it can make it through the adapter stack as an XML message. 

  Also ensure the “Node encoding” is set to String as the default is “XML”

• Start your application and you are good to go

 

• Up until this point, I have not modified the namespace of my schema and everything worked well as expected

 

 

• However, the purpose of this post is to demonstrate how we can break away from the default namespaces that are generated by the adapter.

• Since I have redeployed the application, the message subscription have also been updated:

 

• Start your application up and it should function with this new namespace and therefore subscription.

 

• The next test is to generate a different version of an SAP IDoc and ensure receiving a different version of the IDoc does not break our application.  If you recall from Post 1 we received an error in this situation.  From the Consume Adapter Service wizard, I am deliberately going to generate a schema that has a DOCREL  greater than my current SAP version.

• My subscription is updated and is looking for a “710” IDoc where as SAP is going to generate a
  “700” IDoc

• I received, processed and delivered the message downstream without issue

• Even though SAP, populated the Control record with DOCREL = 700, I am able to process this message because this DOCREL is not being used to derive the target namespace like it is when you use the XML Receive pipeline.  This is really where the value of the Flat file pipeline comes in as it will use the namespace in the schema that is configured inside the pipeline instead of having the Adapter generate a namespace automatically.   This also works as the structure of the 700 IDoc has not changed in the 710 IDoc.  If the structure of the IDoc changes with the version number then you have no other option than to regenerate your schemas.

 Gotcha!

So everything is fine and dandy until you try to deploy another IDoc with the same Program Id/Partner Profile.  There is bug with the BizTalk Adapter Pack that prevents you from deploying multiple IDoc Flat File schemas with the same Program ID.  Look for a workaround with my next post in this series.  I should be able to turn that post around quicker than this one.

Off Topic: Canada Wins....Mikael loses

During this past MVP summit there was an abundance of trash talk regarding Olympic Hockey. Mikael Hakanssons and I made a Blog bet with the loser of the bet dedicating a post to the winning country. As I am sure you have heard, Canada beat USA in overtime 3-2 on Sunday. I will give Mikael credit for being a good sport even though he jumped on the USA bandwagon as soon as Sweden was eliminated. You can find his post here.

Consuming a BizTalk hosted WCF Service with InfoPath 2007

I recently built a POC that wrapped an SAP RFC call in a WCF Service exposed by BizTalk. I exposed schemas as WCF services instead of opting for exposing the Orchestration as a WCF Service. The purpose of the Service was to validate a utility meter’s status inside of SAP.  Some of the statuses may include active/inactive/meter not found.  The RFC that I used is called RFC_Read_Table.  This is a pretty generic, out of box,  RFC that allows you to query SAP tables.  The RFC is fairly complex due to a lack of good documentation available.  Perhaps I am just not part of the secret SAP consultant’s club where you can actually get useful documentation.  I plan on documenting my findings in an upcoming post so that other people do not have to endure the pain I went through in order to get it working.

 

One of the benefits of InfoPath is the ability to consume a web service (asmx).  I was pleasantly surprised that InfoPath can also consume some  WCF endpoints(.svc).  The endpoint type that I exposed was based on the basicHttpbinding which happens to be one of the more interoperable bindings.  At this point, the SOAP adapter has been identified as an Adapter that will be removed in an upcoming release of BizTalk.  The idea behind this is that people should be moving to the WCF adapter.  There are some gaps between the SOAP and WCF adapter, which is outside the scope of this post, but my advice would be if you can use the WCF Adapter then you should.  If you cannot, due to a limitation, then you should provide this feedback to Microsoft so they can either enhance the WCF adapter or hold off on removing the SOAP adapter.

 

The InfoPath service consumption wizard is fairly straight forward with the exception of  “how the form works with the Web Service”.  The 3 options that you do have include:

• Receive and submit data
• Submit data
• Receive data

 

When I think of a client consuming a Web Service, I think in terms of the client providing a Request message and the Server providing a Response.  With this in mind, I decided to use the “Submit data” option.  What I ran into is that I could submit data to my BizTalk Service, BizTalk would process the request and provide a response but nothing was being updated in my form.  I discovered that I needed to use the “Receive and submit data” option instead.  The description is counter intuitive as it implies I am receiving data first then I am submitting it.  Perhaps if Microsoft would have called this option “Submit and receive data” this would have been a little clearer.

 

I have recently been teased by a few colleagues and friends about the amount of detail and screen shots that I have put in my blog posts.  At the risk of more torture,   here is a step by step guide of how I got this all working.

• Design a Form Template

 

• Select “Web Service'” and click OK

 

• If you want to submit data to a service and use the response data in your form, select “Receive and submit data'”

 

 

• Provide the destination of your WSDL and click “Next”

• The name of my operation is “ValidateMeter”, click “Next”

• Thus far, the information that has been provided is related to the Web Service response so I have provided a name for this data connection that reflects this.

• Since I am submitting data to this same service, I need to provide the WSDL URL again.

• Once again I need to provide the operation that I will be communicating with.

• The “ValidateMeter” operation has a single input parameter.  I need to specify which field from the request schema will be sent to the Web Service.  To do this I want to click on the following button:  

 

• Select the field from Schema

• Click “Next” to proceed

• I now need to provide a name for this data connection

 

• At this point I have a blank canvas and I also have the ability to start dragging and dropping fields from my Web Service Request and Response schemas onto the canvas.

 

• I have expanded, the “queryFields” node and have dragged the “MeterNumber” element onto the canvas.  If you recall the value that will be populated in this field will be sent as part of the Request message to the ValidateMeter Service.

• When BizTalk provides the response from SAP, I want to display this value on the form.  In order to do this, I need to expand the “dataFields” node and select “MeterStatus” from the Web Service Response schema.

 

• To test my form, I can click on the “Preview” button

• I can now provide a Meter Number and click the “Run Query” button.  Within a couple seconds I will see my Meter Status text get populated.  In this particular instance, the Meter Number 123456 does not exist so the Meter Status text box is updated to reflects this.

BizTalk 2009: BizTalk Mapper - get value from last node

I ran into a situation where I needed to get the last value from a re-occurring node that SAP was returning.

In order to extract the last record, I utilized the Record Count functoid and fed that value into the Index functoid. The Index functoid is also fed with the value from the "WA" element that I am really interested in. The result is then passed to a String Concatenate functoid where the data is enriched before being assigned to the destination document.

BizTalk Adapter Pack 2.0 – SAP Adapter IDoc Schema Versioning Part 1

A colleague and I have been working through some anomalies when it comes to SAP IDoc schemas that were generated using the BizTalk Adapter Pack 2.0 SAP Adapter. This is the adapter which is based upon WCF and NOT the .Net Connector based adapter.

When you go to generate an IDoc using the Consume Adapter Service wizard in Visual Studio, you will be presented with a screen like this. You will most likely have a few options when selecting an IDoc and version to use. To be sure that you have selected the right version of IDoc, you should contact your BASIS admin.

As you select different versions of IDocs, notice that the Node ID changes. This Node ID will soon become your Target Namespace in the XSD schema that you are about to generate.

This Target Namespace is very important since BizTalk uses the Target Namespace and root node to determine subscriptions.

If you have chosen the wrong version of IDoc you will soon know. You will receive a subscription error indicating that you have received a message that BizTalk does not have a Schema for.

Event Type: Error

Event Source: BizTalk Server 2009

Event Category: BizTalk Server 2009

Event ID: 5719

Date: 1/17/2010

Time: 1:54:31 PM

User: N/A

Computer: Server

Description:

There was a failure executing the receive pipeline: "Microsoft.BizTalk.DefaultPipelines.XMLReceive, Microsoft.BizTalk.DefaultPipelines, Version=3.0.1.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" Source: "XML disassembler" Receive Port: "WcfReceivePort_SAPBinding_IdocZHR_KPV3R710_Custom" URI: "sap://CLIENT=XXX;LANG=EN;@a/SAPSERVER/XX?ListenerGwServ=SAPGWXX&ListenerGwHost=SAPSEVER& ListenerProgramId=XXXXXX&RfcSdkTrace=False&AbapDebug=False" Reason: Finding the document specification by message type "http://Microsoft.LobServices.Sap/2007/03/Idoc/3/ZHR_KP//700/Receive#Receive" failed. Verify the schema deployed properly.

If you refer to the first image in this blog post, you will see that the Node Id has a value of http://Microsoft.LobServices.Sap/2007/03/Idoc/3/ZHR_KP//710/Receive#Receive where as BizTalk has received an IDoc with a Namespace of http://Microsoft.LobServices.Sap/2007/03/Idoc/3/ZHR_KP//700/Receive#Receive. So how can this be? SAP by default generates an IDoc based upon the current release version. But you also have the ability to send a specific IDoc version as configured in the SAP Program ID. This provides some flexibility in the event you need to use a legacy IDoc version.

When SAP sends an IDoc out, it will populate this version information in the control record. I have turned on tracking on the receive port and as you can see this information has been populated by SAP.

Since I am using an XML Receive pipeline and SAP has no idea what namespace BizTalk is requiring, the SAP Adapter is generating this namespace at run-time based upon the DOCREL and IDOC_VERSION elements.

The lesson here is to make sure you know what version of the IDoc is currently configured with your Program Id within SAP. I must also caution that while this approach does work, it may not live through your next SAP upgrade. At least not without updating your BizTalk projects. With each SAP major release, you can expect this DOCREL value and potentially the IDOC_Version to increase. So the next time you upgrade your SAP environment, you may get into the same situation that I have described(subscription errors) and it may take some significant refactoring to get your BizTalk schemas and maps back in sync with your SAP version.

In the next post of this series, I will discuss how you can use a Receive Pipeline and Flat File Pipeline Disassembler to override this namespace so that it does not become dependant upon the DOCREL and IDOC_VERSION elements…stay tuned.

ShareTalk Integration (SharePoint/BizTalk) – Part 10 Adding Metadata to SharePoint columns in a messaging only solution

I had a requirement where I needed to upload images to a SharePoint library and use the information from the image filename to populate columns within the document library. Performing these tasks within an orchestration is pretty trivial as you can set context properties in a Message Assignment shape that will drive the behavior of the WSS Adapter.  I couldn’t justify performing these tasks in an Orchestration since it would involve an extra hop to the MessageBox in order for Orchestration to be invoked.

Another option is to do all of this work in a Pipeline component.  I can create  a Send Port subscription that would allow me to access the context properties from the message received and update them to include the WSS properties all within in a pipeline. This would allow the solution to become a pure messaging solution and I can save an extra Message Box hop.  When building this solution, I referenced Saravana Kumar’s white paper on Pipeline Components.  It came in handy,  especially in the area of creating the Design time pipeline properties.

Within the Pipeline Component, the first thing that I wanted to to was retrieve the source file name.  I am able to do this by reading the “ReceivedFileName” property from the File Adapter’s context properties. I then wanted to clean this file name up since the value of this property includes the entire UNC path: \\Servery\RootFolder\SubFolder\filename.jpg.  I have written some utility methods to parse the file name from this long string.

There are 3 parts to the image file name(1213455_NEW_20100110120000.jpg) that I am particularly interested in.  They include an Asset Number, Asset State and the Date/Time that the image was taken.  The scenario itself is a field worker who needs to capture an image of an asset, indicate the Asset Number and its state i.e. New/Old.  This information is then captured in the name of the image.  Since it is an image, there is no other reasonable way to store this meta data outside the file name.  This in itself is the reason why I need to to use a pipeline component.  Standard WSS adapter functionality includes the ability to use XPATH statements to extract data from the message payload and populate a SharePoint document library column. 

Once I have captured this meta data and massaged it to my liking, I want to then provide this context data to the WSS Adapter.  The WSS adapter is a little different than most of the other adapters in that you can populate an XML document and push that into the ConfigPropertiesXml context property.  The document structure itself is a flat XML structure that uses a “key-value” convention.

<ConfigPropertiesXml><PropertyName1>Column 1 Name</PropertyName1><PropertySource1>Column 1 Value</PropertySource1><PropertyName2>Column 2 Name</PropertyName2><PropertySource2>Column 2 Value</PropertySource2><PropertyName3>Column 3 Name</PropertyName3><PropertySource3>Column 3 Value</PropertySource3></ConfigPropertiesXml>

 

I also want to populate the WSS Adapter’s Filename context property.  I can achieve this by the following statement:

pInMsg.Context.Write("Filename",
        "http://schemas.microsoft.com/BizTalk/2006/WindowsSharePointServices-properties", ImageFileName);

 

Below is my Execute method, in my pipeline component, where all of this processing takes place.  You can download the entire sample here.  This code is at a proof of concept stage so you will want to evaluate your own error handling requirements. Use at your own risk.

public IBaseMessage Execute(IPipelineContext pContext, IBaseMessage pInMsg)
{
    //Get received file name by retrieving it from Context
    string FilePath = pInMsg.Context.Read("ReceivedFileName", "http://schemas.microsoft.com/BizTalk/2003/file-properties") as string;

    //strip path from filename
    string ImageFileName = GetFileNameFromPath(FilePath);

    //Utility methods to parse filename
    string msgAssetNumber = GetAssetNumberFromFileName (ImageFileName);
    string msgAssetState = GetAssetStateFromFileName(ImageFileName);
    string msgImageDateTime = GetImageDateFromFileName(ImageFileName);

    //Write desired file name to context of WSS Adapter
    pInMsg.Context.Write("Filename",
        "http://schemas.microsoft.com/BizTalk/2006/WindowsSharePointServices-properties", ImageFileName);

    // Populate Document Library Columns with values from file name
    string strWSSConfigPropertiesXml = string.Format("<ConfigPropertiesXml><PropertyName1>{0}</PropertyName1><PropertySource1>{1}</PropertySource1>" +
        "<PropertyName2>{2}</PropertyName2><PropertySource2>{3}</PropertySource2><PropertyName3>{4}</PropertyName3><PropertySource3>{5}</PropertySource3></ConfigPropertiesXml>",
        this.AssetNumber,msgAssetNumber,this.AssetState,msgAssetState,this.ImageDateTime,msgImageDateTime);

    pInMsg.Context.Write("ConfigPropertiesXml", "http://schemas.microsoft.com/BizTalk/2006/WindowsSharePointServices-properties", strWSSConfigPropertiesXml);
    return pInMsg;
}

A feature that I wanted to provide is the ability to provide the SharePoint column names at configuration time.  I didn’t want to have to compile code if the SharePoint team wanted to change a column name.  So this is driven from the pipeline configuration editor.  The values that you provide (on the right hand side) will set the column names in the ConfigPropertiesXml property that is established at run time.

 

If you provide a value in this configuration that does not correspond to a column in SharePoint, you will get a warning/error on the Send Port.

Event Type:    Warning
Event Source:    BizTalk Server 2009
Event Category:    (1)
Event ID:    5743
Date:        1/10/2010
Time:        7:38:19 PM
User:        N/A
Computer:   
Description:
The adapter failed to transmit message going to send port "SendDocToSharePoint" with URL "wss://SERVER/sites/BizTalk%%20Repository/Inbound%%20Documents". It will be retransmitted after the retry interval specified for this Send Port. Details:"The Windows SharePoint Services adapter Web service encountered an error accessing column "Missing Column" in document library http://SERVER/sites/BizTalk%%20Repository/Inbound%%20Documents. The column does not exist. The following error was encountered: "Value does not fall within the expected range.".

This error was triggered by the Windows SharePoint Services receive location or send port with URI wss://SERVER/sites/BizTalk Repository/Inbound Documents.

Windows SharePoint Services adapter event ID: 12295".

For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

 

The end result is that I can use information contained in the file name to populate meta data columns in SharePoint without an Orchestration.

Also note, I have built this pipeline component so that it can be used in a Receive Pipeline or as Send Pipeline by including the CATID_Decoder and CATID_Encoder attributes.

 

   [ComponentCategory(CategoryTypes.CATID_PipelineComponent)]
   [ComponentCategory(CategoryTypes.CATID_Decoder)]
   [ComponentCategory(CategoryTypes.CATID_Encoder)]
   [System.Runtime.InteropServices.Guid("9d0e4103-4cce-4536-83fa-4a5040674ad6")]
   public class AddSharePointMetaData : IBaseComponent, IComponentUI, IComponent, IPersistPropertyBag 
  

 

ShareTalk Integration (SharePoint/BizTalk) – Part 9 Overwrite - Rename Mode

While further investigating SharePoint-BizTalk integration capabilities I ran into a file overwrite mode called “Rename” which I thought was rather unique.  I have never seen something like this in the FILE or FTP adapters so I figured that I would post something on it.

The WSS Adapter has an Overwrite property  that allows you select one of the following modes:

The Yes and No values are pretty self explanatory.  Orchestration allows you to determine which mode you want to set  within an Orchestration at run time.

When you select the Rename value, BizTalk will use the static file name that you have configured in the Send Port configuration or the actual file name of the source document.  If the document library does not contain a file with this name already, it will be added with the correct filename.

 

Where it gets interesting is when you try to insert a file with the same file name as one that already exists in the same document library.  When this situation occurs the first portion of the file name will remain intact, however a unique identifier will be added to the end of the filename.  BizTalk will not raise a warning or error indicating that this event has occurred.

So when would I use this?  I think this feature could be used in one of those “exception to the rule cases” where the business has indicated that “there will never be a duplicate file uploaded into this document library” but it actually happens.  It is also helpful when the view of the document library has been configured in such a way that the documents are sorted by file name.  This way it makes it easier to identify one of these cases as opposed to just using a Message ID where the document could end up being displayed in a very random fashion.