Introducing Windows Azure BizTalk Services Preview –Part 1

 

On Monday, June 3rd Microsoft announced the Preview of Windows Azure BizTalk Services (aka WABS or just BizTalk Services).  Windows Azure BizTalk Services is a managed service provided by Microsoft.  More specifically you can think of BizTalk Services as a Platform as a Service (PaaS) offering.  Recently, I had an opportunity to work with this platform as I was involved in an NDA event in New York City.  While I was preparing for that presentation I took the opportunity to document some of my learnings so that I could share them when the content was no longer considered NDA.  With the TechEd announcement this information is now considered public hence the timing of this blog post.

The purpose of this offering is to provide customers a way to execute integration scenarios in a cloud based environment.  The idea is that you would design and build your interfaces locally within Visual Studio and then deploy your interface to the cloud where it is executed.

Conceptually we can think of a Windows Azure BizTalk Services much like any other interface that utilizes an Integration platform.  We have source systems, an intermediary where we can perform functions like Message Validation, Enrich, Transformation and Enrich (aka VETE).  We then have the ability Route our message to the appropriate destination system.  The difference with this platform is that the “heavy lifting” is occurring in the Windows Azure cloud as opposed to our On-Premise BizTalk Servers.

 

There are some subtle differences between Windows Azure BizTalk Services and BizTalk Server 2013 (On-Premise) solutions.  I say subtle because for an experienced BizTalk person these new concepts are not that big of a leap.  Below is s diagram containing some of the key artifacts and components that make up a BizTalk Service solution.

Bridges

The term that jumps out at you is probably Bridges. What is a Bridge? A Bridge is a message processing entity.  Probably the closest BizTalk construct that you can relate a Bridge to is an Orchestration but there are differences.  The reason why Orchestration is “kinda close” is that it groups a series of action or processes together within one logical container.

Bridges include Sources, Pipelines and Destinations.  Currently within the Windows Azure BizTalk Services Preview we have 3 types of Bridges:

• Xml One-Way Bridge
• Xml Request-Reply Bridge
• Pass-Through Bridge (for message transport)

Within a Pipeline we can call transformations, perform validations, encode, decode or enrich messages.

We also have 3 types Source types:

• FTP/S
• HTTP
• SFTP

and 9 Destination types:

• FTP
• One-Way External Service Endpoint (No Service Bus connectivity)
• One-Way Relay Endpoint (Service Bus connectivity)
• Service Bus Queue
• Service Bus Topic
• SFTP Destination
• Two-Way External Service Endpoint
• Two-Way Relay Endpoint (Service Bus connectivity)

Technically this Destination list is not fully complete.  We also have the ability to communicate with On-Premise LOB systems which I will go through in another Blog Post.

SDK Installation

Even though our BizTalk Services run in the cloud, we still perform our development activities within Visual Studio.  Currently Visual Studio does not ship with the appropriate templates that allow us to build BizTalk Services solutions.  However, we can get these templates from the Windows Azure BizTalk Services SDK.

The installation is pretty straight forward and I have included some screen shots below.

Note: On 64 bit machines only need to install the 64 bit SDK

 

 

 

Visual Studio Experience

When we launch Visual Studio 2012 we will discover a new C# templates under the BizTalk Services node:

If we choose BizTalk Service we see a blank diagram that allows us to drop a Bridge and Source or Destination endpoints onto it.

If we choose BizTalk Service Artifacts we will see a new project that does not include a bridge but a Transformation (map) and two sample schemas.

If we choose the BizTalk Service Project template we have more capabilities as we can drop any of the following Artifacts/Actions onto our canvas.

 

In the image below I have modeled a scenario just to visualize what building an interface in BizTalk Services may look like.  In this scenario we are a company that specializes in managing Request For Information (RFI) processes. Organizations that have projects that they would like vendor responses to will send us these requests and we will publish this information to our trusted vendor network that allow them to be informed of these opportunities. 

In this scenario I have an XML One-Way Bridge which will expose an HTTP(s) Endpoint.  Within this bridge we will perform an transformation from the message type that was sent by our customer requesting RFI information into a message type that all of our Vendor partners are expecting. Once we have transformed this message, we need to drop it on a Service Bus topic. 

Much like a BizTalk Server project we will start with our schemas.  This schema editor is very similar, if not the same, as the BizTalk Server schema editor. The first schema that we will build represents our client request.

 

Next we will create a schema that we use with our strategic vendors who subscribe to our service. You will notice that there are more fields in the partner schema.  This is where the value of our firm comes from as we have direct communication with client and have some additional insight into their project about whether the project is likely to proceed and will also expose the name of the customer.  Typically these values would be looked up from a CRM system but for now we will keep it simple and include it in the map.

Next we will add a Transformation(or Map) to our solution that will allow us to transform our CustomerRequest into a PartnerRequest message.

Getting deep into the mapper is outside the scope of this post but you will see some new “functoids” or what are now being called Message Operations.  Some of these new capabilities include ForEach, MapEach, If-Then-Else, List Operations and YES you can call custom code.

Once we have defined our Schemas and Maps we can now configure our XML Bridge.  Within this bridge we will specify our message type that we will be receiving (CustomerRequest).  We will also specify our Xml Transform (Map).  Since we are not dealing with flat files and do not have any Validation requirements we will leave those stages in their defaulted stage.

We now need to wire up the output of our Bridge to our Destination endpoint which is a Service Bus Topic.  We will do so by selecting a Connector operation and then connect the dots(kinda like in Viso) between our Bridge and Topic endpoint.

 

Now remember that we can perform message routing in BizTalk Services.  To set our Routing patch we will need to click on the line that we drew from our Bridge to our Topic.

We now need to click on the Filter Condition.  In this case we are not going to restrict any messages from continuing to our topic so we will select Match All but we do have the ability to create a filter based upon message content.

Next we will change our focus to the Service Bus Topic.  We need to configure this endpoint by clicking on it and then modifying the PartnerTopic.config (filename will vary) file for it.  Within this config file we will specify our Service Bus namespace, topic name,Issuer Name and Key.  Please note that these are production Service Bus entities and have nothing to do with the BizTalk Services Preview environment.

We also want to rename our XMLOneWayBridge so that we are not using the default name since this name will makeup part of our URI. 

Lastly we want to specify our endpoint where our Bridge can be contacted. To do this click on any empty space on the canvas and then modify the Deployment Endpoint.  (Note the endpoint below is fictitious).

At this point we can now deploy our solution to Windows Azure BizTalk Services.  We can do so by right mouse clicking on our Visual Studio Solution and selecting Deploy Solution. Here is where we want to specify our BizTalk Services environment configuration.  You cannot use your Production Windows Azure credentials here.

The deployment only takes a few seconds and when it is completed we should see a message like the following:

Testing

Part of the SDK that we just installed includes utilities that can be used when Sending and Receiving messages to/from BizTalk Services.  We are going to go ahead and leverage the MessageSender application.

The Message Sender Application is a command line tool that is expecting the following inputs:

• ACS Namespace (for BizTalk Services environment)
• Issuer Name
• Issuer Key
• Bridge Endpoint Address
• File location to message you want to send
• Content Type

The result is that our message has been sent to our bridge successfully.

If we go to the Windows Azure Production portal we will discover that we have a message within our Subscription.

Conclusion

If some of this looks familiar it is because it is!.  This is the evolution of what use to be called Windows Azure EAI/EDI Labs.  It is now being branded underneath the “BizTalk” brand which makes sense on a few levels in my opinion.  For instance the team that is working on this new service is the same team responsible for the BizTalk Server product.  One thing you will hear the product team speak to is that there is “ONE BIZTALK”. One platform with many capabilities including BizTalk Server On-Premise, BizTalk Server running as Infrastructure as a Service (IaaS) and now BizTalk Services which is running as a Platform as a Service offering.

There is another huge aspect to BizTalk Services that has been neglected from this post as I just am not familiar with it but EDI in the cloud also makes up a substantial part of this offering and arguably provides the greatest area of opportunity.

I am just getting started with blogging about BizTalk Services.  Stay tuned for my next post on how to integrate On-Premise Line of Business Applications(SAP)  with BizTalk Services.

BizTalk 2013: WCF-SAP Adapter using Messaging Servers

 

SAP supports a few different architectures when it comes to scaling out.  In some environments, mainly smaller environments, the components that facilitate sending and receiving messages (such as IDOCs, BAPIs or RFCs) may be co-hosted on an Application Server.  In some larger environments a dedicated Messaging Server(s) may be provided. We can also use a named destination that we define in our saprfc.ini file.

Up until this point all of my experience integrating with SAP has occurred through an Application Server.  This is the default Connection Type that is populated in the Send Port configuration of a WCF-SAP send port.

Recently I ran into a situation where the SAP environment was not using Application Servers.  Instead a Messaging Server was being used.  The end result was that I received the following error

 

A message sent to adapter "WCF-SAP" on send port "SendGetSAPEquipment" with URI "sap://CLIENT=010;LANG=EN;@b/*SAPSERVER*/*ENV*?RfcSdkTrace=False&AbapDebug=False" is suspended.
Error details: Microsoft.ServiceModel.Channels.Common.ConnectionException: Details: ErrorCode=RFC_OK. ErrorGroup=RFC_ERROR_COMMUNICATION. SapErrorMessage=Connect to message server failed
Connect_PM  MSHOST=*MSHOSTValue*, R3NAME=*ENV*, GROUP=PUBLIC

LOCATION    CPIC (TCP/IP) on local host with Unicode
ERROR       service *SAPSERVER* unknown
TIME        Tue Apr 23 09:38:18 201
RELEASE     700
COMPONENT   NI (network interface)
VERSION     38
RC          -3
MODULE      nixxhsl.cpp
LINE        776
DETAIL      NiHsLGetServNo: service name cached as unknown
COUNTER     1.  AdapterErrorMessage=. ---> Microsoft.Adapters.SAP.RFCException: Details: ErrorCode=RFC_OK. ErrorGroup=RFC_ERROR_COMMUNICATION. SapErrorMessage=Connect to message server failed
Connect_PM  MSHOST=*MSHOSTValue*, R3NAME=*ENV*, GROUP=PUBLIC

Microsoft.ServiceModel.Channels.Common.ConnectionException: Details:
ErrorCode=RFC_OK. ErrorGroup=RFC_ERROR_COMMUNICATION.
SapErrorMessage=Connect to message server
ailed
Connect_PM  MSHOST=*MSHOSTValue*, R3NAME=*ENV*, GROUP=*MyOrganization*

Since the SAP environment was actually using Messaging Server(s), In order to successfully make a connection to SAP, here is the correct configuration that I needed to provide:

• Application Server Group Name
• Message Server Host
• R/3 System Name
• Connection Type B

These properties will be specific to your environment and therefore I have blacked them out.  You will need to get this information from your BASIS admin.

 

However, this is not enough and this is where I got hung up.  You also need to specify TCP port where SAP will be looking for these connections.  This property is not exposed through the BizTalk GUI.  In order to specify this TCP port you need to modify the following file:

C:\Windows\System32\drivers\etc\services

Within this file you will want to add an entry like:

MSHOST            1234/tcp   # SAP ENV System Message Server Port

Once again these values will be specific to your environment.  I have added fictitious values here.  You may see something like SAPMS<ENV> but once again your mileage may vary.  In all likelihood your TCP Port will not be 1234 (that I made up).  It will likely be 4 digits though.

The most important thing to recognize is that the value in this file (that I have called MSHOST) must match the value that we see in the error message below.  This value will not match any values within your Send Port.

 

It isn’t only BizTalk that will use these values.  Your SAPGUI will also use these same values when connecting to SAP systems.  So if you are stuck, my recommendation is to find a machine that is running the SAPGUI and then copy over those values to the file that exists on your BizTalk Server(s).  I have no idea how this MSHOST value is looked up as the BizTalk Send Port has no knowledge of it. My guess is that at runtime the R3Name in combination of the Message Server Host is somehow used to lookup this MSHOST from this file.

 

Conclusion

Hopefully this blog post will save you some time.  I know that I had a tough time finding information that clearly painted the picture for me...especially in the area of the port# that is found in our services file.

Presentation from recent talks

In the past week I have had the opportunity to speak at two events.  On May 1st I spoke at the Phoenix Connected Systems User Group and on May 6th I gave two presentations at the BizTalk Bootcamp in Charlotte.

It was a lot of fun putting these demos together and I received a lot of good feedback so hopefully you will also benefit from them.  You can find the slide deck here: http://sdrv.ms/12fBU53

Note: Some of the Service Bus slides were borrowed from Clemens Vasters’ deck which he has made freely available here.

Upcoming Speaking Engagements

I am happy to announce a couple of upcoming speaking events:

May 1st – Phoenix Connected Systems User Group in Tempe, Arizona

Introducing BizTalk 2013 and the new Cloud adapters

Microsoft recently released their 8th version of BizTalk called BizTalk Server 2013. One of the new capabilities in this version is the ability to use out of the box adapters when communicating with Windows Azure Service Bus endpoints and other Software as a Service (SaaS) solutions. In this session Kent will focus on exposing Line of Business services to the cloud using the new WCF-BasicHttpRelay, SB-Messaging and WCF-WebHttp adapters. This session will include some slides and lots of demos including exposing SAP information to a mobile device using these technologies.

May 6th/7th  - BizTalk Boot Camp in Charlotte, North Carolina

BizTalk 2013 - New cloud-based BizTalk Server adapters

This session will also focus on the new Cloud based adapters for BizTalk Server 2013.  I will be demonstrating some of the new capabilities found in the BasicHttpRelay, SB-Messaging and WCF-WebHttp adapters.

BizTalk 2013 CU1 has been released

The product team recently announced that BizTalk 2013 CU1 has been released. Within the CU there are 3 issues that are fixed:

1. User cannot perform certain database-related operations in BizTalk Server 2013
2. BAM tools cannot be configured in a multi-node BizTalk Server 2013 environment
3. The vertical scroll bar on the target schema does not work correctly when
   you use Visual Studio to design a BizTalk Server 2013 map

Personally, the only issue that I have noticed was item number 2.  It is nice to see this fixed as I was having a horrible time configuring BAM in a multi-server environment.  I definitely recommend installing the update.  This time around, Microsoft is delivering these updates via the Windows Update process.  Overall, the process is painless and rather convenient.  Windows Update will not automatically install these updates, they are something that you need to “opt in” for.

To install the update:

• Launch Windows Update

• Check for updates

• You should discover a BizTalk 2013 section as part of the Important Updates.  If you want additional info, click on the More Info link on right hand side.

 

Note: The KB article indicates A reboot is not required, however I was prompted for a reboot but that must be related to the Windows Server Updates.

BizTalk Monitoring Whitepaper is now available

For the past month and half, in my spare time, I have been working on a whitepaper that compares and contrasts System Center Operations Manager (SCOM) and BizTalk360. 

I have used both products extensively in my professional life and in order to remain objective within the paper I have driven my evaluation off of requirements.  These requirements are based upon my experience of managing the 24x7 operations of a BizTalk environment at a large Power Distribution company in Canada.

The paper can be downloaded from here and feel free to leave any comments that you may have below.

I would also like to thank Steef-Jan Wiggers for being a technical reviewer on this paper.  He provided me with some valuable feedback that increased the quality of the paper.

 

BizTalk 2013 has reached RTM and what it means for Customers

 

Microsoft has reached another important milestone with the release of their 8th version of BizTalk.  This is the 8th release in 13 years!!!  No major product in Microsoft has shipped that often so my congratulations goes out to the team that shipped this release.   Microsoft made this announcement on Thursday, March 21 on the BizTalk Server Team Blog. Many other members within the BizTalk community have also shared this news via their blogs, Twitter and Facebook.  Here are a few examples:

• Codit – General Announcement
• Stephen Thomas – How to find the bits on MSDN
• Ben Cline – General Announcement
• Nino Crudele – General Announcement
• BizTalk 360 and the licensing changes

Since many of these blogs, and especially the Product group blog, get into the actual features I will not re-post those since you have likely already read it.  What I did want to get into is why someone should make the jump to this version whether you are a new BizTalk customer or a customer looking to upgrade.

Back in December 2012 I started working with a new organization who was just getting started with using integration platforms to solve their integration requirements. The green light was given to proceed with implementing BizTalk.  The next question was which version do we go with?  BizTalk 2010 was very mature at this point and was the version that I was using at my previous employer.  On the flip side BizTalk 2013 had been in CTP since July 2012 and a subsequent Beta was released in early November.  I spent a lot of time with these versions(see my blog posts from this timeframe) and after speaking with the product group at the BizTalk Summit in December we all agreed that BizTalk 2013 was the way to go. 

With the help of the product group, we were able to get this organization, that I am working for, into the TAP program.  The TAP program gave me access to early bits and allowed me to start building out the various Dev, Test and Prod environments.  It also gave me access to Product Group members like Guru, Rajesh and Michael who ensured I had what I needed to make the progress required to hit my project’s deadline.  Thanks Guys :-)

My environments are now up, I have the new SCOM management pack installed and we are just finishing up User Acceptance Testing (UAT) before we are officially live.  Going live will occur in mid-April.

If I look at the drivers for going with BizTalk 2013, here are a few of them:

• The new cloud based adapters(SB-Messaging and WCF-WebHttp) - Something that was very important to my stakeholders is having a flexible integration platform.  They do not have the appetite to spin up many types of integration platforms to perform integration.  Having one platform to deal with on-premise integration and another to deal with Cloud or SaaS just wasn’t an option.  One of the demos that I did build for the organization when we were deciding on BizTalk was to take an existing SAP interface and expose it on premise.  I then took that same interface and exposed a RESTful interface via the cloud (ServiceBus) using the new capabilities of BizTalk 2013.  Another example platform flexibility is this organization does use some Cloud/SaaS applications from large well known providers.  Having an Integration tool that can easily integrate with these solutions was a must have.  Between the core WCF adapters and the new WebHttp adapter I feel very confident that BizTalk can satisfy the integration requirements when interfacing with these Cloud/SaaS applications.
• SFTP Adapter – This initially was not a big motivator but it has paid off.  In the past I have used the /nSoftware adapter to satisfy Secure FTP requirements.  The /nSoftware adapters are good and I have blogged about them quite a bit in the past.  When we were going through the BizTalk 2013 evaluation there were no SFTP requirements for this company.  Fast forward 3 months and 2 SFTP requirements have popped up when dealing with external partners.  In the end this turned out very well.  I didn’t have to go back to the ‘well’ to ask for more money to buy 3rd party licenses.  Instead this is an out of the box adapter which allowed me to build these SFTP interfaces very quickly.  Instead of dealing with a procurement process, I can focus on solving the problem.  That brings real value to customers.
• Improved ESB Toolkit integration – I have used the ESB Toolkit in the past, but more from an Exception Management portal perspective.  I support any improvements that simplify the installation and reduce the amount of time it takes to deploy.  In BizTalk 2013, Microsoft has simplified the ‘core installation’ of the ESB Toolkit.  This is a MUCH better experience and I applaud their efforts.  I still think there is an opportunity to take this one step further though.  Simplify the installation of the Exception Management Portal.  I know this is designated as a ‘sample solution’ that you can adopt for your own use but why not improve the experience and give customers a complete experience?  Another area of improvement is updating the Alerting Service that will look for exceptions in the Exceptions Database and then send alerts to people who have subscribed to them from the Exception management portal.  This Alerting service is extremely useful but is currently broken as part of this release.  The Alerting Service has a dependency on Microsoft Enterprise Library 4.1 and the core ESB Toolkit is using Microsoft Enterprise Library 5.0.  I haven’t had the time to dig in and find a resolution but my advice would be to include both the Exception Portal and Alerting service as part of the core product and then wrap the same, simplified installation that has been provided for the core framework.
• Improved SharePoint Adapter experience – In a prior life, I did a lot of integration between BizTalk and SharePoint.  I even wrote a couple chapters on it in the BizTalk LOB book.  One of the unfortunate realities of the previous SharePoint adapter is that you had to install an adapter web service on SharePoint servers.  As you can imagine, the people who are responsible for SharePoint within organizations are not thrilled with this approach.  As someone who is walking into a new organization, you are not going to make a lot of friends if you need to install ‘BizTalk bits’ on their servers. In SharePoint 2010, the Client Object Model was introduced to allow developers to programmatically add/modify/remove content from SharePoint (in addition to many other things).  BizTalk has now adopted this same model and is leveraging this Client Object Model within the Adapter.  The Adapter Web Service still exists to allow for backwards compatibility during migrations but you can now use the SharePointAdapter and specify using the Client Object Model instead.  Another benefit of using the Client Object Model approach is BizTalk is now able to use this same adapter to communicate with Office 365.  This new organization that I am working with also has some plans to use Office 365 in the future so I am anticipating that the Client Object Model will be leveraged in the near future.  I also won’t have to beg the SharePoint team to install the ‘BizTalk bits’ on their on-premise SharePoint servers.
• BizTalk Infrastructure as a Service (IaaS) – This is a feature that was included as part of the Beta and will be included as part of the core offering as well.  While I am not going to go out on a limb and say that I will leveraging this for production right away but I like to have the option.  Having recently gone through the process of creating Dev, Test and Prod environments it can be time consuming.  By time consuming, I am not referring to the actual BizTalk installation and configuration.  Many other time consuming tasks need to take place including procuring servers (either physical or virtual), storage, Active Directory groups and accounts, SQL Server instances etc.  Having the ability to spin up a BizTalk environment in Azure in minutes is attractive.  For my needs, I can see myself leveraging these type environments in Project settings where you can’t disrupt the current Dev/Test/Prod Landscape but we will see how this progresses.  One thing is certain though, having options is good.

Conclusion

There are a lot more features that are included in this BizTalk 2013 release than I have described here.  I thought it would be interesting to describe some of my thought process when going through the BizTalk 2010 vs BizTalk 2013 decision.  Now that BizTalk 2013 has RTM’d this decision gets even easier.  I am very glad that I did proceed down the 2013 route as otherwise I would be missing out on a lot of great features.

Also, stay tuned for some more announcements from the BizTalk team.  They are working on something BIG that fits into the Platform as a Service (PaaS) space. Hint – it is related to this.  Once again having options is good and the service that will be offered provides customers more options when it comes to cloud integration.  BizTalk developers will be happy as a lot of their existing skillset can be leveraged while building out some new capabilities using this upcoming platform.  Stay tuned – it is worth it.