Dynamics AX 2012 R2 File based integration with BizTalk Server 2013

 

I am currently involved in a project where I need to integrate Dynamics AX with a 3rd party payroll system.  Dynamics AX 2012 provides a rich integration framework called Application Integration Framework or (AIF).  One of the core strengths of this framework is the ability to generate a typed schema through a configuration wizard.  Combine that with a AX’s ability to create inbound and outbound ports and you now have the ability to generate export(and import if interested) files rather quickly.

When I mentioned in the previous paragraph “typed messages” I meant that AX will generate XSD schemas that we can include in our BizTalk projects.  This is a breath of fresh air compared to some other ERP systems where you get handed a CSV flat file that you have to build a flat file schema for.

In my scenario I was receiving a list of Work Orders so a colleague working on the AX side was able to provide me with the Work Order Schema and an imported schema that includes AX types.  At this point I add the schemas into my solution, build my map and wire everything up.  Go to run an initial test and was presented with the following error:

Details:The published message could not be routed because no subscribers were found. This error occurs if the subscribing orchestration or send port has not been enlisted, or if some of the message properties necessary for subscription evaluation have not been promoted.

This is a pretty standard error message that basically comes down to BizTalk received a message that it was not expecting.  The reason why BizTalk was not expecting it was because AX wraps outbound messages with a SOAP Envelope as you can see in the image below.

 

SOAP Envelopes are certainly nothing new but I didn’t expect AX to use them when writing a file to the file system.  When receiving Web/WCF Services BizTalk automatically takes care of extracting the message body from the incoming SOAP message for us.  With the FILE Adapter that facility just does not exist.

You will notice in screenshot below that there is a namespace that is specific to AX.  This got me thinking that AX probably has an XSD for this message type as well.

After digging around a bit I did find the location of the AX schemas to be in the Program files\Microsoft Dynamics AX\60\Server\MicrosoftDynamicsAX\bin\Application\Share\Include folder.  The schema that I was looking for was called Message.xsd

Just adding this the BizTalk project was not enough.  I needed to make a few small tweaks to the schema:

• Click the “Schema Icon” of the schema and then set the Envelope property to be True.  This instructs BizTalk that it is an envelope schema and when BizTalk sees this message that it needs to strip out the Envelope which in this case is a SOAP Envelope.

• Set the Body XPath property by selecting the Root Node of the schema and then populating the appropriate value which in this case is

/*[local-name()='Envelope' and namespace-uri()='http://schemas.microsoft.com/dynamics/2011/01/documents/Message']/*[local-name()='Body' and namespace-uri()='http://schemas.microsoft.com/dynamics/2011/01/documents/Message']/*[local-name()='MessageParts' and namespace-uri()='http://schemas.microsoft.com/dynamics/2011/01/documents/Message']

We can now deploy our application. When it comes to our Receive Location that will be picking up this message, we want to ensure that we are using the XMLReceive Pipeline.  Within this Pipeline the XML Disassembler stage will take care of the SOAP envelope so that when the message body is presented to the MessageBox that  any subscribers will receive the expected message body.

Conclusion

When I first discovered that I was receiving a SOAP wrapped message my instincts said maybe AX could just use a WCF port instead of a FILE port.  This just wasn’t the case, there are only two options when it comes to configuring an outbound port: FILE and MSMQ.  Using MSMQ would not of helped me in this case as the same issue would have existed. 

AX certainly does provide the ability to call a WCF service but it is a more custom based approach.  I would have had to expose this schema as a WCF service but then my AX colleagues would have had to write code against the proxy to populate all of the different data elements.  This would have defeated the purpose of using the AIF framework in order to expedite the ability to delver a solution under very tight timelines.  Luckily with a little tinkering we were able to come up with a reasonable solution without writing custom code.

I have to think that AX is wrapping these messages in a SOAP Envelope for a reason.  Perhaps a WCF outbound port is coming in an upcoming release?

Windows Azure BizTalk Services Preview (Part 2) –BizTalk Adapter Services SAP Integration

 

Over the past 7 years I have spent a lot of time integrating BizTalk and SAP systems.  I probably built more interfaces for SAP than any other system.  When new technology such as Windows Azure BizTalk Services surfaces I am always interested to understand what the experience looks like.

If you read my prior post in this series, you are probably wondering how it is possible to use Windows Azure BizTalk Services in the Windows Azure Cloud and connect to your On-Premise LOB Systems like SAP?  Enter BizTalk Adapter Services.

BizTalk Adapter Services allows us to use the BizTalk Adapter Pack in order to connect to LOB systems.  The BizTalk Adapter Services are hosted in IIS and leverage the Service Bus Relay (under the hood) in order to traverse Firewalls and NATs.  Using the BizTalk Adapter Service is just another tool in our toolbox when achieving Hybrid integration.

Adapter Services Installation

I am now going to quickly install the BizTalk Adapter Services and then provide a walkthrough of how we can integrate with SAP.  The scenario that we are going to walk through is part of a larger process that I won’t get into much detail within this blog but we have an SAP process were we need to feed sub-systems within the organization with Equipment data from SAP.  We will get this equipment by calling a custom SAP RFC(BAPI).

The BizTalk Adapter Service is a component that we want to install on a Server within our Network.  This machine does not require BizTalk Server to be present.

 

This is an important step as the account that we specify here will require access to SQL Server where configuration will be stored about our Relay Endpoints.

In this case I am using a local SQL Server instance.  You can use SQL Express if you so desire but I have leveraged SQL 2008 R2 (SQL Server 2012 is also supported).

This step kinda reminds me of the BizTalk SSO Service password but they are not related or linked.

The BizTalk Adapter Service installation will be installing a Management Web Service.  Since I am just using this for demo purposes I did not enable SSL. 

Once we have finished installing the BizTalk Adapter Service we will discover a new database called BAService.

We will also discover a new entry in Server Explorer within Visual Studio called BizTalk Adapter Services. 

If we right mouse click on BizTalk Adapter Services we have the ability to add a BizTalk Adapter Service by specifying the URL of the Management Service that was created as part of the installation process.

Once this Adapter Service has been added we can expand it and discover our LOB Targets.

We can add a new SAP target by right mouse clicking on SAP  and selecting Add SAP Target.

We now need to specify our SAP connection information much like we would if we were adding Generated Items within a BizTalk Server 2013 solution.  We need to specify the Application Server, the System Number, Client and Language.  I recently blogged about connecting to SAP Messaging Servers.  Not to worry these more advanced scenarios are supported by clicking on the Advanced button.

Next we need to provide our SAP credentials if we are not using Windows Credentials.

Much like the BizTalk Server experience we can search operations including IDOCs, BAPIs and RFCs.  Do note that at this point we are only able to push messages to SAP.  This means we can send an IDOC and call a BAPI/RFC and get the response back.  SAP cannot push IDOCs or call an RFC that is hosted by the BizTalk Adapter Service. 

Once we have found the artifact that we are looking for we can add it to the Selected operations list and then click the Next button.

Once again need to provide credentials that the BizTalk Adapter Service will use to connect to SAP.

This gets a little interesting.  Much like in my previous blog post where we connected to a Service Bus Topic, we will use the production Service Bus connectivity information.  So in this case I am going to leverage an existing Service Bus Namespace and credentials to enable Service Bus Relay capabilities.

We also have the ability to specify a Target sub-path that just allows us to organize our endpoints.

We now have a confirmation page that we can just click the Create button in order to finish the wizard.

With our SAP LOB Target created we can now drag that Target onto our canvas:

At this point we have just created an SAP target and added it to our canvas. but we have not actually generated our schemas that will be used within our solution.  In order to generate these schemas we need to right mouse click on our SAP LOB Target and specify Add Schemas to Project.

We will once again be prompted for SAP credentials.

Within our solution, we will now find two schemas have been added.  One is the core schema and the other is a schema for data types leveraged by the core schema.

A Web Request Schema, that we will expose externally to calling applications, is also needed.  This schema is very simple with only two fields: Plant and EquipmentName.

Next we need to add a Map that will allow us to transform our Web Request message into our SAP Request message.

The map is pretty straight forward as we only need to map a couple fields.

A Map that will allow us to transform our SAP Response into our Web Response is also required.

Next, we need to add  an Xml Request-Reply Bridge to our canvas by dragging it onto our canvas.

We now want to modify the Property page for this artifact  as it will become part of our overall URI.

By double clicking on our  Bridge, we have the ability to configure it.  Within this configuration we will want to specify the Message Type for the message that we will be receiving from the calling application and the Message Type for the type of message we will be sending back to the calling application.

We also want to specify our two Transformations.  The first one will take our Web Request and map it to our SAP Request.  The second Transformation will take our SAP Response and map it to our Web Response.

Once our Bridge has been configured, we need to connect our Request Response Bridge to our SAP LOB Target.  We can do so by dragging a Connector from our Toolbox and connecting our Bridge to our SAP LOB Target.

Routing messages is a core functionality within BizTalk Services. In order to route messages, we need to provide a filter condition that will allow us to specify messages to flow between the Bridge and the SAP LOB Target. In order to set this filter we need to click on the line that connects these two artifacts and then click on the Filter Condition.

Since we want all messages to flow to this LOB system we will choose Match All and click the OK button.

We also need to specify a SOAP Action much like we need to do with BizTalk Server.  In order to do this we need to highlight the connection between the Bridge and LOB target and then click on Route Action.

In this case we want to specify a SOAP Action and then specify the action from our Request Schema.  Much like the EAI/EDI Labs solutions we want to wrap our Expression around single quotes( ‘   ‘)

We are now almost ready to deploy our solution but before we can do that we need to specify our Deployment Endpoint.  We can do so by clicking on any blank space on our canvas and then specifying our Deployment Endpoint address in the Property page.

Deploying our solution is as simple as right mouse clicking on our Visual Studio Solution file and selecting Deploy Solution.

In order to deploy our solution we will need to provide the Deployment Endpoint, ACS Namespace(for BizTalk Service environment), Issuer Name and Issuer Shared Secret.

Within a few seconds our solution will be deployed and will be ready for testing. Once again I can  use the MessageSender application that is part of the BizTalk Services SDK. But, since looking at a black screen with white xml text isn’t very appealing, I created an ASP.Net Web application and borrowed the MessageSender class from the SDK project.  In this case I have my Web Application running on a laptop that isn’t on the same network as the machine that is hosting the BizTalk Adapter Service.

 

Within the web application, we have the ability to provide a Plant and a wild card string representing a piece of equipment in order to get more details about where that piece of equipment has been deployed, the Manufacturer and Model Number.

 

Conclusion

Hopefully this walkthrough has provided you with some valuable insight on what it takes to integrate a Line of Business system like SAP with Windows Azure BizTalk Services.  I can certainly envision scenarios where an organization uses an LOB system like SAP but may not have a dedicated integration platform to perform some of this integration. Windows Azure BizTalk Services may fill this void and enable some business scenarios that just may not have been possible for that organization.  Consider another situation where you may have a SalesForce system that requires SAP data (maybe Equipment data ) and this type of functionality in BizTalk Services really becomes a business catalyst.

I also think that the BizTalk Adapter Service may provide some interesting Mobile use cases as well as we can now expose SAP data through the cloud in a secure manner.

While the scenario is a bit dated, I also wrote a similar article on how to submit IDOCs to SAP using the Windows Azure EAI/EDI Labs CTP which has now been superseded by Windows Azure BizTalk Services.

Introducing Windows Azure BizTalk Services Preview –Part 1

 

On Monday, June 3rd Microsoft announced the Preview of Windows Azure BizTalk Services (aka WABS or just BizTalk Services).  Windows Azure BizTalk Services is a managed service provided by Microsoft.  More specifically you can think of BizTalk Services as a Platform as a Service (PaaS) offering.  Recently, I had an opportunity to work with this platform as I was involved in an NDA event in New York City.  While I was preparing for that presentation I took the opportunity to document some of my learnings so that I could share them when the content was no longer considered NDA.  With the TechEd announcement this information is now considered public hence the timing of this blog post.

The purpose of this offering is to provide customers a way to execute integration scenarios in a cloud based environment.  The idea is that you would design and build your interfaces locally within Visual Studio and then deploy your interface to the cloud where it is executed.

Conceptually we can think of a Windows Azure BizTalk Services much like any other interface that utilizes an Integration platform.  We have source systems, an intermediary where we can perform functions like Message Validation, Enrich, Transformation and Enrich (aka VETE).  We then have the ability Route our message to the appropriate destination system.  The difference with this platform is that the “heavy lifting” is occurring in the Windows Azure cloud as opposed to our On-Premise BizTalk Servers.

 

There are some subtle differences between Windows Azure BizTalk Services and BizTalk Server 2013 (On-Premise) solutions.  I say subtle because for an experienced BizTalk person these new concepts are not that big of a leap.  Below is s diagram containing some of the key artifacts and components that make up a BizTalk Service solution.

Bridges

The term that jumps out at you is probably Bridges. What is a Bridge? A Bridge is a message processing entity.  Probably the closest BizTalk construct that you can relate a Bridge to is an Orchestration but there are differences.  The reason why Orchestration is “kinda close” is that it groups a series of action or processes together within one logical container.

Bridges include Sources, Pipelines and Destinations.  Currently within the Windows Azure BizTalk Services Preview we have 3 types of Bridges:

• Xml One-Way Bridge
• Xml Request-Reply Bridge
• Pass-Through Bridge (for message transport)

Within a Pipeline we can call transformations, perform validations, encode, decode or enrich messages.

We also have 3 types Source types:

• FTP/S
• HTTP
• SFTP

and 9 Destination types:

• FTP
• One-Way External Service Endpoint (No Service Bus connectivity)
• One-Way Relay Endpoint (Service Bus connectivity)
• Service Bus Queue
• Service Bus Topic
• SFTP Destination
• Two-Way External Service Endpoint
• Two-Way Relay Endpoint (Service Bus connectivity)

Technically this Destination list is not fully complete.  We also have the ability to communicate with On-Premise LOB systems which I will go through in another Blog Post.

SDK Installation

Even though our BizTalk Services run in the cloud, we still perform our development activities within Visual Studio.  Currently Visual Studio does not ship with the appropriate templates that allow us to build BizTalk Services solutions.  However, we can get these templates from the Windows Azure BizTalk Services SDK.

The installation is pretty straight forward and I have included some screen shots below.

Note: On 64 bit machines only need to install the 64 bit SDK

 

 

 

Visual Studio Experience

When we launch Visual Studio 2012 we will discover a new C# templates under the BizTalk Services node:

If we choose BizTalk Service we see a blank diagram that allows us to drop a Bridge and Source or Destination endpoints onto it.

If we choose BizTalk Service Artifacts we will see a new project that does not include a bridge but a Transformation (map) and two sample schemas.

If we choose the BizTalk Service Project template we have more capabilities as we can drop any of the following Artifacts/Actions onto our canvas.

 

In the image below I have modeled a scenario just to visualize what building an interface in BizTalk Services may look like.  In this scenario we are a company that specializes in managing Request For Information (RFI) processes. Organizations that have projects that they would like vendor responses to will send us these requests and we will publish this information to our trusted vendor network that allow them to be informed of these opportunities. 

In this scenario I have an XML One-Way Bridge which will expose an HTTP(s) Endpoint.  Within this bridge we will perform an transformation from the message type that was sent by our customer requesting RFI information into a message type that all of our Vendor partners are expecting. Once we have transformed this message, we need to drop it on a Service Bus topic. 

Much like a BizTalk Server project we will start with our schemas.  This schema editor is very similar, if not the same, as the BizTalk Server schema editor. The first schema that we will build represents our client request.

 

Next we will create a schema that we use with our strategic vendors who subscribe to our service. You will notice that there are more fields in the partner schema.  This is where the value of our firm comes from as we have direct communication with client and have some additional insight into their project about whether the project is likely to proceed and will also expose the name of the customer.  Typically these values would be looked up from a CRM system but for now we will keep it simple and include it in the map.

Next we will add a Transformation(or Map) to our solution that will allow us to transform our CustomerRequest into a PartnerRequest message.

Getting deep into the mapper is outside the scope of this post but you will see some new “functoids” or what are now being called Message Operations.  Some of these new capabilities include ForEach, MapEach, If-Then-Else, List Operations and YES you can call custom code.

Once we have defined our Schemas and Maps we can now configure our XML Bridge.  Within this bridge we will specify our message type that we will be receiving (CustomerRequest).  We will also specify our Xml Transform (Map).  Since we are not dealing with flat files and do not have any Validation requirements we will leave those stages in their defaulted stage.

We now need to wire up the output of our Bridge to our Destination endpoint which is a Service Bus Topic.  We will do so by selecting a Connector operation and then connect the dots(kinda like in Viso) between our Bridge and Topic endpoint.

 

Now remember that we can perform message routing in BizTalk Services.  To set our Routing patch we will need to click on the line that we drew from our Bridge to our Topic.

We now need to click on the Filter Condition.  In this case we are not going to restrict any messages from continuing to our topic so we will select Match All but we do have the ability to create a filter based upon message content.

Next we will change our focus to the Service Bus Topic.  We need to configure this endpoint by clicking on it and then modifying the PartnerTopic.config (filename will vary) file for it.  Within this config file we will specify our Service Bus namespace, topic name,Issuer Name and Key.  Please note that these are production Service Bus entities and have nothing to do with the BizTalk Services Preview environment.

We also want to rename our XMLOneWayBridge so that we are not using the default name since this name will makeup part of our URI. 

Lastly we want to specify our endpoint where our Bridge can be contacted. To do this click on any empty space on the canvas and then modify the Deployment Endpoint.  (Note the endpoint below is fictitious).

At this point we can now deploy our solution to Windows Azure BizTalk Services.  We can do so by right mouse clicking on our Visual Studio Solution and selecting Deploy Solution. Here is where we want to specify our BizTalk Services environment configuration.  You cannot use your Production Windows Azure credentials here.

The deployment only takes a few seconds and when it is completed we should see a message like the following:

Testing

Part of the SDK that we just installed includes utilities that can be used when Sending and Receiving messages to/from BizTalk Services.  We are going to go ahead and leverage the MessageSender application.

The Message Sender Application is a command line tool that is expecting the following inputs:

• ACS Namespace (for BizTalk Services environment)
• Issuer Name
• Issuer Key
• Bridge Endpoint Address
• File location to message you want to send
• Content Type

The result is that our message has been sent to our bridge successfully.

If we go to the Windows Azure Production portal we will discover that we have a message within our Subscription.

Conclusion

If some of this looks familiar it is because it is!.  This is the evolution of what use to be called Windows Azure EAI/EDI Labs.  It is now being branded underneath the “BizTalk” brand which makes sense on a few levels in my opinion.  For instance the team that is working on this new service is the same team responsible for the BizTalk Server product.  One thing you will hear the product team speak to is that there is “ONE BIZTALK”. One platform with many capabilities including BizTalk Server On-Premise, BizTalk Server running as Infrastructure as a Service (IaaS) and now BizTalk Services which is running as a Platform as a Service offering.

There is another huge aspect to BizTalk Services that has been neglected from this post as I just am not familiar with it but EDI in the cloud also makes up a substantial part of this offering and arguably provides the greatest area of opportunity.

I am just getting started with blogging about BizTalk Services.  Stay tuned for my next post on how to integrate On-Premise Line of Business Applications(SAP)  with BizTalk Services.