BizTalk integration with SharePoint and the Cloud Presentation

I recently had the opportunity to speak at the Calgary .Net User Group. My presentation included slideware and demonstrations involving integrating BizTalk with SharePoint 2010 and the AppFabric Service bus. You can view the presentation below:

BizTalk 2010 Beta – Where are the AppFabric Service Bus bindings?

After installing the Windows Azure platform AppFabric SDK V1.0 - April Update I was expecting all of the AppFabric Service Bus relay bindings to be available inside of the BizTalk Admin Console but they were not.  I could run the AppFabric Echo .Net Sample without any issues.  After re-installing both the AppFabric SDK and BizTalk 2010 Beta I was still unable to find these bindings in the Admin console.  I checked the machine.config files (both 32 bit and 64bit) and both files were updated.  The problem with this is that I checked the .Net 2.0 config files.

After stumbling across Wade Wegner’s blog, the picture became much clearer.  Wade explains “I soon realized that the issue was the machine.config file.  When you install the Azure AppFabric SDK the relevant WCF extensions are added to the .NET Framework 2.0 machine.config file, which is shared by .NET Framework 3.0 and 3.5.  However, .NET Framework 4.0 has its own machine.config file, and the SDK will not update the WCF extensions.”

To resolve the issue you need to run a utility called: RelayConfigurationInstaller.exe which can be found in C:\Program Files (x86)\Windows Azure platform AppFabric SDK\V1.0\Assemblies (64 bit machine) or C:\Program Files\Windows Azure platform AppFabric SDK\V1.0\Assemblies (32 bit machine) and provide an input file that includes

<?xml version ="1.0"?>
<configuration>
<startup>
<requiredRuntime safemode="true"
imageVersion="v4.0.30319"
version="v4.0.30319"/>
</startup>
</configuration>

The name of this file should be called RelayConfigurationInstaller.exe.config.

example:

C:\Program Files (x86)\Windows Azure platform AppFabric SDK\V1.0\Assemblies>RelayConfigurationInstaller.exe /i

After this command was run successfully, I could now see the “Cloud Bindings” including the NetTcpRelayBinding

Presenting at Calgary .Net User Group - June 22, 2010

Topic: BizTalk integration with SharePoint and the Cloud
Speaker: Kent Weare
Date: 22-June-2010
Location: Nexen Conference Center
801-7th Ave. S.W., Calgary, AB. (Plus 15 level)
Map
Registration: 5:00 pm - 5:30 pm
Presentation: 5:30 pm

http://dotnetcalgary.com/


Abstract
Part 1 - Discover how you can leverage BizTalk to integrate with SharePoint and InfoPath

In this session, Kent will demonstrate some of the ways that you can integrate with SharePoint using the BizTalk WSS Adapter and Web Services. Kent will also discuss some real world scenarios where this technology has been used to provide value to the business.

Part 2 - Discover how you can leverage BizTalk to integrate with the AppFabric Service Bus

Learn about Microsoft's cloud based Service Bus and how you can leverage BizTalk to communicate with external applications and services. Kent will demonstrate BizTalk's ability to expose on premise services to the cloud and consume AppFabric Service Bus end points. Kent will also discuss some scenarios where it may be appropriate to leverage these integration patterns in the real world.

TechEd 2010 – Day 4

TechEd 2010 North America is done for another year.  I thought the conference went really well and people were full of enthusiasm.  A lot of Product Groups have shipped, or are shipping products this year.  There were over 11 300 in attendance for this event eager to hear about the latest bits in the pipeline .  I was at TechEd in LA last year and can tell you there wasn’t near that many people in attendance.

 

BizTalk and the Cloud

I saw a really interesting session where BizTalk was brokering a connection through the Service Bus into a SAP and a Mainframe system.  A Web app was hosted in Azure and BizTalk was listening for requests via the AppFabric Service Bus.  It was probably the riskiest demo I saw, but Chris and Elizabeth pulled it off without a hitch.  I have been thinking about these types of scenarios quite a bit since I have done a fair about of integration with SAP in the past.  SAP certainly cannot connect into the Azure Service Bus at this time nor do I expect them to be able to do that any time soon.  Mainframe systems are no different since they most likely will not have a transport mechanism capable of communicating with the Microsoft cloud.  Many businesses leverage SAP to run their core business, having BizTalk capable of connecting into the cloud provides an additional layer of abstraction and security when accessing SAP data.

 

More Cloud

There was certainly no shortage of Cloud talk at this conference.  The next session I saw was called Appfabric: Extend .Net technologies to the Cloud.  It was presented by Aaron Skonnard and Keith Brown.  You can tell these guys are very comfortable on the presenter’s stage.  It was probably best presentation that I have seen in a long time.

Some of the topics discussed include:

• You can think of Azure AppFabric as an extension of the .Net framework
• No Azure Workflow as a Service in this release
• Relay bindings allow your application access into the Service Bus
• Direct connections allow you to communicate peer to peer with other clients in order to improve performance and reduce charges
• Eventing pub/sub can be used in multi-cast situations
• To increase interoperability use Http Relay Bindings

View from the BizTalk Booth

• A had a few conversations with attendees who were using other middleware products such as Oracle or IBM and wanted to know how BizTalk stacked up in terms of feature set.  None of the features that they were requesting were absent from the BizTalk stack.  They also wanted to see how easy the Microsoft tools are to use and the impression that I got is that they felt they were on par with what they were use to seeing.
• Another attendee wanted to know whether they should be using  WF or BizTalk.  Quite frankly this scenario can still get a little blurry for me.  We talked about things like low latency, durable messaging, flat file requirements, Adapters and management of the solutions.

 

Next Year

Next year TechEd will be held in Atlanta from May 16th-19th.  There is currently an early bird special where you can save $500 for registering early.  http://northamerica.msteched.com/?fbid=qjus7wvTc2O

TechEd 2010 – Day 3

More BizTalk

We took a closer look at the Dashboard Settings in the BizTalk Admin Console. There are 3 main categories that you can configure:

• Group Configurations
• Host Configurations
• Host Instance Configurations

Probably the most compelling setting allows you to control the polling interval when connecting to the MessageBox for new messages. You even have the ability to configure these settings based upon a Messaging or Orchestration scenario at the Host level. In the demo, we saw a simple Receive and Respond scenario. The default configuration setting is to have BizTalk connect to the MessageBox every 500 ms. This message took .68 seconds to be processed end to end. The Polling interval was then changed to 10 ms and the message took .09 seconds to process. Some pretty dramatic improvements. The end result is that the BizTalk Host Instances are not waiting as long for their next polling interval so messages will be processed more frequently. These actions are not without some risk though. The more frequently you poll, the more stress you put on you SQL Server database so mileage will vary.

The portability of the Dashboard settings were also demonstrated. You now have the ability to export and import your configuration settings across environments so you are not manually tweaking on several servers. You also have the ability to script the import and export of these settings

The new BizTalk Management Pack for SCOM was demonstrated. They have rebuilt this MP from the ground up and now include 2 separate views:

• Application
• Deployment

The Application view focuses on artifacts that would typically be addressed by BizTalk developers. So if you have a message that failed in a pipeline or a retrying sent port then you would find them in this view. More infrastructure related items would be found in the Deployment view. So if you had a database or host instance offline, you will find these types of alerts in the Deployment view.

Unfortunately, the new SCOM pack will only work with BizTalk 2010 so if you are still on BizTalk 2009 or previous, you cannot use this MP. The good news is that the MP will ship with BizTalk 2010 so there will be no gap between those two products and their release dates.

Other bits

• The support for SQL 2005 and Windows 2003 is gone for BizTalk 2010
• BizTalk 2009 (and earlier) will not be certified for Windows Server 2008 R2.
• Business Activity Monitoring (BAM) Portal will support 64 bit app pools - Not exactly true, please see comment by Rahul for further clarification
• BizTalk Admin Console is still not a native 64bit app. You can run in WOW mode on 64 bit machine though

BizTalk Business to Business integration

I later sat in on a B2B presentation that discussed EDI and Trading Partner Management. I have not had a lot of exposure to these 2 features of BizTalk so it was interesting to get a look under the hood. The TPM management is greatly improved and is more intuitive to use as you can now group Business Units under a Parent entity.

They also discussed the BizTalk Mapper since EDI schemas tend to be large and complex. Leveraging new features like Search, Indicative Match, copy and pasting functiods, highlighting selected nodes and links goes along way.

The enhanced FTP adapter was discussed and here are some of the core features:

• Support for Read Only Receive scenarios
• FTPS protocol is now supported
• Improved performance when connecting to AIX systems
• Atomic Sends for Text files

Communications Server 14

Office Communications Server (OCS) has been rebranded to just Communications Server or “CS 14”. I am not that familiar(at a deep technical level) with OCS and related technologies but since we do use it I thought I would check out the Q and A session that deals with interoperability.

I think it is pretty safe to say that this industry is in a state of transition. You have traditional telecom companies like Cisco and Avaya who are trying to hold onto their footprint and then you have Microsoft who is very hungry in this vertical. The frustrating part is that these companies have so much at stake that they are not collaborating much (from what I can see) and that leaves customers who have a blend of this technology frustrated as it is tough to develop roadmaps with so much changing. It may be a good idea to sit on the sidelines for a year to see how this all shakes out.

View from the BizTalk Booth

More great conversations at the BizTalk booth:

• An Infrastructure Architect wanted to know what impact BizTalk has on his environment since his organization is planning on implementing BizTalk
• Another attendee is looking and managing the provisioning of Datacenter assets through BizTalk. He wanted BizTalk to host services that would expose functions like “deploy Windows Server”, “Deploy SQL Server”, “Deploy Application ‘x’ to Server ‘y’”. It was a very interesting discussion about how they wanted to provision systems to their private cloud.
• More questions about using the BizTalk Mapper activity in WF
• There genuinely seems to be interest in BizTalk. Perhaps people are just being polite, but a lot of people are talking about introducing BizTalk to their environment(s).

TechEd 2010 – Day 2

 

I opened up day two with a session involving Connecting to Line Of Business (LOB) systems in 15 minutes

We saw:

• A demonstration of the BizTalk mapper activity in WF
• Many questions from the audience regarding the licensing model.  Some people questioned that if you had to buy a BizTalk Standard license to use the Mapper and LOB adapters then why wouldn’t you just use BizTalk.  The response was that there may be some scenarios where you still want to avoid the BizTalk message box but want to connect to LOB systems like SAP without having to write a lot of code to transform data between the two systems.  The BizTalk Mapper will easily pay for itself over time as it allows you to convert data through a GUI tool instead of writing a lot of code.
• You will not have to install all of BizTalk in order to use these two new activities in WF.  Microsoft plans on providing a “AppFabric Connect” feature that allows you to just install the BizTalk bits that you need
• Licensing for these components will be based on the same model that BizTalk standard edition uses so you will be paying per proc(and not by the core)
• Oracle adapter exposed as WCF service that SharePoint BCS (Business Connectivity Services) consumed...no code is required.
• Connecting to IBM lob systems in a similar manner.  Use the BizTalk HIS components outside of BizTalk and connect to legacy IBM systems without BizTalk.
• I got the sense that a lot of .Net developers were excited to see these capabilities but not so excited that they needed to pay for BizTalk licenses in order to use these bits.

Another interesting session I saw involved extracting SAP data so that it can be used by Microsoft BI tools.  The tool is called Simplement Data Liberator.  We heard from one of their customers who uses a blend of SAP and Microsoft tools.  They explained some of their Roadblocks to “True BI”;

• SAP tables difficult to understand
• SAP tables difficult to extract
• Need too many experts
• Explicit skill set required to use a SAP BI tool
• Needed real time access to SAP data
• Needed SharePoint integration
• Better value with Microsoft Consultants
• Wanted wider talent pool

SAP has over 50 000 tables so trying to extract data manually or through Custom code is a daunting experience.  The Simplement Data Liberator tool has solved this challenge. The Tool replicates SAP data into SQL Server. You can get both German and English Meta data  and SAP Security  info is also replicated.  This tool eliminates the need to write ABAP for reporting in SAP.  Surface SAP data into Microsoft BI products by using SQL queries.

Other features:

• Work with only tables you need
• Load into SQL server via replication
• Manipulate/Transform data such as dates (ETL)
• Attached meta data so that you can use English Table names
• Use SharePoint, Excel services or SQL Server Reporting Services (SSRS) to query data

We saw a demo where a customer was updated in the SAP GUI system and within seconds (3-5) the data was in the SQL BI system.  I must say that I was impressed by the demo, it seems like this vendor understands SAP very well and has figured out a way to lower customer costs when consuming SAP data.

View from the BizTalk Booth 

More great questions and conversations about BizTalk:

• An excellent conversation about End to End Message Security.  You always want to trust your developers to do the right thing, but how do you prevent a developer from adding their own receive location, or subscription in BizTalk for malicious purposes; to prevent a rogue developer from depositing money into  their Bank Account via BizTalk.  If anyone one has thoughts on this type of scenario and have figured out a way to solve it I would be interested in hearing about it – use comments below.
• Whether BizTalk is truly pub/sub due to the polling nature of the Host instance connecting to BizTalk MessageBox looking for more work.  This attendee was looking for more Event Driven processes inside BizTalk.
• A DBA was interested to know how BizTalk will connect to their Database and some of the “gotchas” to look out for
• A few people that closed the door on BizTalk after BizTalk 2004 that are now back taking another look at BizTalk 2010.  I can’t say this enough, the product has matured significantly over the past and it is worth taking another look at
• Connecting BizTalk to the AppFabric Service bus in order to connect with other external parties

TechEd 2010 – Day 1

Below are some of my ad hoc notes from TechEd 2010 in New Orleans. 

The Atmosphere

Much more encouraging than last year, in my opinion.  The numbers seem to be up and the convention centre is very busy even though the facility is huge.  Microsoft has rented over 1 mile worth of space to host the event.  At a meeting that I attended on Sunday, a Microsoft rep indicated that registrations had exceeded their expectations this year.  They attributed this to improvements in the North American economy.

Keynote

A great deal of the keynote was dedicated to cloud computing.  The phrase that has been floating around Microsoft circles for the last little while is “we’re all in” meaning Microsoft is betting big time on the emergence of cloud computing.

Some items that were discussed include new tools available for Developers and IT Pros to monitor cloud applications including:

• Intelitrace for Azure
• SCOM Management packs for monitoring Web Roles and other artifacts in the cloud
• SQL Management Studio integration for managing SQL Azure databases

Microsoft discussed a client, Chicago Tribune, and their shift from moving On-Premise assets to the Cloud.  Some of the key drivers for this move includes:

• Reduction in the amount of Data centers required to run their operations
• Pay as you go Scalability.  As you can imagine there would be some very newsworthy days that would stress a media outlet’s IT infrastructure
• Reliability and Uptime.  If your site is down, people will just go to your competitor’s site.

Other announcements

• Windows 7 SP1 and Windows 2008 R2 SP1 will enter Beta in July (2010)
• Windows 2008 R2 SP1 will include enhancements such as Dynamic Memory
• Office Communication Server 14 is schedule for Q4 – 2010 Key Features include
• Tighter SharePoint and Exchange integration
• Picture displays in the OCS Client as pulled from SharePoint
• Pictures include a presence border (Green/Yellow/Red)
• Microsoft has over 74 000 Enterprise voice users internally and saves $1 million per month by using OCS and Live Meeting for long distance and conference calls
• Use OCS to search for Skills within your organization
• 720p video supported with OCS 14
• Exchange pushes polices down onto Windows Mobile 7 phones
• Windows Mobile 7 supports multiple exchange accounts including on-premise and in the cloud
• Windows Mobile 7 easily separates your work and you life within one phone
• Edit documents from SharePoint using your Windows Mobile phone

Other Sessions

BizTalk 2010 and VNext

• Customer wants include better alignment with Platform releases (SQL/Windows/VS)
• Session was full – BizTalk interest is alive and well
• Enhanced Mapper exists in 2010 that allows for
• Functoid “Move to page”
• Reduction of noise by focusing on specific areas
• Node Search
• Undo
• New Ribbon at the top of Mapper – Ribbons are in-style
• Mapper and LOB Activities are available within WF Workflows
• A license will be required – potentially a BizTalk Standard license
• Potentially, BizTalk Dev edition may become a free version
• Enhanced BizTalk Management Pack for SCOM will be included
• Better terminology and modeling used
• Better support for multi-node BizTalk deployments
• Adoption of SCOM still low for BizTalk deployments (around 10-20% of the room uses SCOM – non-scientific survey)
• Settings Dashboard available in 2010
• Allows for more granular control over performance settings
• Some Settings remain Global, some Host specific and some Host Instance/Machine specific
• Settings are scriptable via WMI
• Upgrade
• Full support from BizTalk 2006 R2 and BizTalk 2009
• New FTP Adapter
• FTPS protocol is supported
• Support for read only scenarios
• Improved performance when connecting to AIX systems
• VNext
• Plans still vague but looking at more Cloud integration – perhaps we will see BizTalk in the cloud or Transformations as a service in the cloud

Building Communication enabled Business Processes with Microsoft Communications Server 14

This session contained some really cool demos and makes me wish I had more spare time.  I saw some demos where a CRM application was extended to include OCS integration.  You may be saying what is the big deal you can do this today?  What was really neat was that they were able to push the CRM context into OCS so that when you called/IM’d the recipient their OCS window would be expanded to include context about what you were contacting them about.  Imagine you needed to speak to “Bob” about customer “XYZ’s” payment.  Wouldn’t it be nice to include the data that you are looking at as part of the conversation so that Bob does not have to log into the CRM? This is now possible by using the Microsoft Communications Server 14 API.  The API seemed very intuitive and includes XAML components that allow you to drag and drop these components onto your WPF/Silverlight canvas.  I see plenty of opportunities of this type of functionality in Contact Centre or CRM type applications.

View from the BizTalk Booth

This year I am spending some time helping out at the BizTalk Booth.  I thought it would be interesting to post some of the questions or trends that I am seeing while visiting with other attendees. 

• Many people still don’t know what BizTalk is/does.  Once you explain it to them, they usually provide a response similar to “I have a broken down process that is hard to manage and is unreliable, can I use BizTalk in that scenario”?  The answer in many cases is Yes.
• A great question came up regarding BizTalk and SAP Enterprise Services with respect to performance
• Some people expressed frustrations over previous versions specifically some of the management tools in BizTalk 2004.  For those that have been there before and have moved on, I encourage you to come back and take a look at what BizTalk 2010 has to offer.  The tooling, tracking and management of applications has greatly improved.
• Great questions around how do you manage your BizTalk environments?  The answer is SCOM, but you need to invest some time so that you do not create a SPAM engine.  I wouldn’t want to manage a BizTalk environment without a tool like SCOM as it provides great visibility into the Health of your BizTalk application. Tune, tune and then tune again to find the right balance.