Sunday, March 14, 2010

BizTalk Adapter Pack 2.0 – SAP Adapter IDoc Schema Versioning Part 2

In my first post, I discussed how the BizTalk Adapter pack Consume Adapter Service wizard tightly couples the IDoc schema version with the version of SAP that you used to generate the schema with.  The goal of this post is to discover how you can avoid this tight coupling and hopefully survive an SAP upgrade.  I know the organization that I work for would not be pleased with having to update all of our IDoc schemas when we do our next SAP Upgrade just because the version of SAP has been incremented.

When the BizTalk Adapter Pack came out I was pretty excited that I could throw away my Flat File schemas and pipelines.  While true, it comes at a cost.  When using the XML Receive pipeline with the new Adapter, the Adapter will use data in the SAP Control record to determine what version of the message you retrieving from SAP and will assign a namespace to your message, that contains SAP version information, before depositing it in the message box. If have not changed your SAP version since generating your IDocs you are in business.  If you have upgraded your SAP system or have generated the wrong schemas for your SAP version then you will be in a lot of pain. 

To get around this tight coupling issue, you can revert back to the old style of IDoc processing which involves flat files and pipelines.  Now there is nothing really wrong with this model, but it does require a few extra steps and is not as “clean” a solution as using the XML Receive pipeline.

I will now take you through how you generate a flat file schema that can be used to process different versions of the same IDoc.  When I say different schemas this means that the version, and therefore namespace, are different but structurally the documents are the same.  Note that I will not go through all steps required to generate an IDoc schema, only those that are important to generate a Flat File schema.  To see a more comprehensive walk through of generating an IDoc schema, please see my Webcast.

 

  • When using Consume Adapter Service wizard, ensure that “GenerateFlatFileCompatible” is set to true.

image

When you set the GenerateFlatFileCompatible property to true, you will now see the flat file specific data within the Schema specification.

image

Without setting GenerateFlatFileCompatible ,  the schema will not contain this flat file information.

image

image

  • You also want to indicate the “ReceiveIDocFormat” will be a String.  You will find out why this is important later on in this post.

image

  • Once the schemas have been imported into Visual Studio, enable “Flat File Extensions” to the “core schema”.  When using the BizTalk Adapter Pack Consume Adapter Service wizard you will generate more than 1 schema unlike the old .Net Connector version where you would always only generate 1 schema.  By “core schema” I mean the one that contains the “meat” of your schema, not one that contains the shell(all of the imported schemas) or any of the base type schemas.

image

  • You will now need to create a Receive Pipeline and add a Flat File disassembler

PipelineComponentDesigner

  • Add your “shell” schema to the document schema property

image

  • Build and Deploy your application
  • When configuring your SAP Receive location you will want to specify your Receive Pipeline as opposed to XML Receive

image

  • On the Binding tab, set the “ReceiveIdocFormat” to String

image

  • On the Messages tab, provide an XPath expression that will extract the flat file data so it can be disassembled by the Receive Pipeline.  The XPath expression that I used is:

    /*[local-name()='ReceiveIdoc']/*[local-name()='idocData']

    The adapter is essentially wrapping this flat file in XML tags so that it can make it through the adapter stack as an XML message. 

    Also ensure the “Node encoding” is set to String as the default is “XML”

image

  • Start your application and you are good to go

 

  • Up until this point, I have not modified the namespace of my schema and everything worked well as expected

image 

 

  • However, the purpose of this post is to demonstrate how we can break away from the default namespaces that are generated by the adapter.

image

  • Since I have redeployed the application, the message subscription have also been updated:

 image

  • Start your application up and it should function with this new namespace and therefore subscription.

 

  • The next test is to generate a different version of an SAP IDoc and ensure receiving a different version of the IDoc does not break our application.  If you recall from Post 1 we received an error in this situation.  From the Consume Adapter Service wizard, I am deliberately going to generate a schema that has a DOCREL  greater than my current SAP version.

image

  • My subscription is updated and is looking for a “710” IDoc where as SAP is going to generate a
    “700” IDoc

image

  • I received, processed and delivered the message downstream without issue

image

  • Even though SAP, populated the Control record with DOCREL = 700, I am able to process this message because this DOCREL is not being used to derive the target namespace like it is when you use the XML Receive pipeline.  This is really where the value of the Flat file pipeline comes in as it will use the namespace in the schema that is configured inside the pipeline instead of having the Adapter generate a namespace automatically.   This also works as the structure of the 700 IDoc has not changed in the 710 IDoc.  If the structure of the IDoc changes with the version number then you have no other option than to regenerate your schemas.

image

 Gotcha!

So everything is fine and dandy until you try to deploy another IDoc with the same Program Id/Partner Profile.  There is bug with the BizTalk Adapter Pack that prevents you from deploying multiple IDoc Flat File schemas with the same Program ID.  Look for a workaround with my next post in this series.  I should be able to turn that post around quicker than this one.

12 comments:

hxbergen said...

Here the hotfix for your problem:

http://support.microsoft.com/kb/977528

After installing this hotfix (only neccessary on developing machine) you have to regenerate the idoc schemas with flatfile extension.

Kent Weare said...

Thanks for spoiling my next blog post hxbergen :-). Yes, I am aware of this hotfix as we logged the issue with Microsoft before the hotfix was released.

hxbergen said...

We have also logged this issue to
Microsoft, and got this hotfix.

Stanley said...

Would this solve the problem where BizTalk is processing a certain IDoc type for instance ORDERS05 with 2 different SAP system that has different IDoc structure? (Difference in max looping and value enums) Hence we cannot deploy just 1 ORDERS05.xsd.
In short how to resolve the issue of deploying 2 ORDERS05 xsds (the meat of the IDocs) without conflict?

Kent Weare said...

Stanley,

It should resolve that issue providing you have different subscriptions for each IDoc. BizTalk will use the combination of Namespace + root node to determing how to route the message(to either orchestration or send port). If this combination is different for each IDoc then you could deploy both since the structures are different.

If the message structures are different then BizTalk should be able to determine how to disassemble the messages in a receive pipeline. The one thing that I am not sure off is whether these structures will be different "enough" or will BizTalk get confused in the pipeline when trying to determine which Schema to use when disassembling.

Stanley said...

Ken,
Thanks for the quick response.
We have 4 separate SAP systems BizTalk integrate with (all with same release number, 620, and IDoc version), hence the issue.
I am now changing the core target namespace for each IDoc ORDERS05.xsd schema and is praying that BizTalk and the new SAP Adapter will not throw a fit. If I do not change the "meat" ORDERS05.xsd target namespace, I think Biztalk could not differentiate one ORDERS05 IDoc from another, since both have the same namespace and root node.
I see you are changing the target namespace of the "shell" receive xsd schema, correct? Do you believe I should not change the IDoc "meat" schema target namespace too? Just changing the IDoc "shell" schema's target namespace suffice?
Many thanks!

Stanley said...

Ken,
I hope you can help...
Why is the WCF inbound body path expression set as you did? Do I have to follow or alter that?

I am getting this error.. do not know why.

There was a failure executing the receive pipeline: "Energizer.BizTalk.Test.Pipelines.IDocReceive, Energizer.BizTalk.Test, Version=1.0.0.0, Culture=neutral, PublicKeyToken=97b55a5e39b0d07e" Source: "Flat file disassembler" Receive Port: "IDocReceive" URI: "sap://CLIENT=;LANG=EN;@a//01?ListenerGwHost=&ListenerGwServ=sapgw01&ListenerProgramId=USA_BIZTALK&RfcSdkTrace=False&AbapDebug=False" Reason: Unexpected data found while looking for:
'E2LFA1M001'
The current definition being parsed is idocData. The stream offset where the error occured is 524. The line number where the error occured is 2. The column where the error occured is 4.

Kent Weare said...

Stanley,

Did you set the node encoding property to "String"? This setting is right below the XPath expression

Kent

Stanley said...

Finally solved my issues... I was using the WCF-Custom instead of WCF-SAP adapter. I have not only changed the "shell" schemas' target namespace, I also changed the "meat" IDoc schema's target namespace and everything works using WCF-SAP.

Ken,
Would you possibly be doing a Part 3 of this? How to do the same on the Send side to SAP.

I am interested to know what that would entails... especially what to set the Outbound WCF message body's Template to, which I assume is neccesary.

Thanks again! Ken

Stan

Kent Weare said...

Hi Stanley,

I still have another post that deals with receiving IDocs but I will try to post a part 4 that deals with the Send side.

Kent

Tom said...

our company is no customer of SAP.

But one partner company of us uses SAP Systems to generate shipment document like "SHPMNT03".

They transfer the files to us using FTP.

I cannot use the SAP-WCF adapter for BizTalk to generate the schema as there SAP team is not ready to open the access for us.

is there a way I can consume the idoc flat file without the WCF SAP Adapter.

Kent Weare said...

If they won't open up access then your only option is to manually create a version of this schema which may be quite challenging.

You can use the Flat File Schema wizard which may speed up some of the development time.

Do you know if this is standard IDOC that they are using? If so I may be able to help you out with getting a copy. Email me: wearsy At gmail.com