BizTalk 2013 Beta: Azure Service Bus Integration–Dynamic Sent Ports

 

I started writing this blog post before the Beta was announced and it was probably a good thing that I did not publish it as I now have something else to discuss.  Microsoft has made an enhancement to the configuration that supports Dynamic Send Ports.  More on this later.

This blog post is really about 3.5 years in the making.  You are probably saying what, how is that possible?  BizTalk 2013, nor the Azure Service Bus(in its current form) did not even exist back then.  This is true but a colleague and myself were already thinking of a cloud based pattern to solve a ‘trading partner’ scenario that we were dealing with.

Business Problem

I work in the Utilities industry within a province in Canada.  We have a ‘deregulated’ market when it comes to Electricity and Natural Gas.  What this means is that the market is more open and it provides other opportunities for companies to participate within it.  This deregulation occurred over 10 years ago and the overall goal was to reduce prices to the end consumer.  (The jury is still out on that).

The market is primarily broken up into 4 segments:

• Generation – Organizations who generate electricity via coal, gas, wind, solar etc
• Transmission – High voltage Transmission of energy from Generation plants to Distribution Networks
• Distribution – Organizations that take High voltage feeds and distribute low end voltage to customer’s homes and businesses.  These Distribution businesses are also responsible for capturing  Customers’ usage and providing this information to retailers.
• Retailers – Are responsible for billing customers for their usage.  Customers can choose who their retailer is but cannot choose who their Distribution company is as it is broken up by Service territory due to the amount of capital investment required in order to provide infrastructure.

Note: Image is used for illustrative purposes only and does not necessarily reflect the actual number of companies involved.

As you can probably imagine, there is a need for information to be exchanged amongst these parties.  Amongst the Generation, Transmission and Distribution companies, each company must be aware of demand that their customers are requesting so that the Generation company can produce enough energy.  The Transmission and Distribution companies must also be able of supporting that load on their networks.

From a Distribution standpoint, the segment of the industry that I work in, we need to be aware of Customer management/Enrollment, Energizing and De-energizing customers,  Meter reading and providing consumption data to Retailers so they can bill the customers.

A few years ago the industry was looking to replace its Managed File Transfer solution that facilitated the exchange of this information.  Think of an FTP type application that had an Address book with some scheduling type activities.  This tool moved around flat files that were all based upon different transaction types.  The tool at the time suffered from major reliability issues

At the time, the Service Bus just wasn’t what it is today.  Also if you think there is a lot of skepticism  about the cloud today, it was much, much worse 3.5 years ago.  People were not ready for the cloud back then and as an industry we went a different direction, I thought it would be fun to revisit this problem and look at a way that we could solve this problem today using cutting edge tools like the Azure Service Bus and BizTalk Server 2013.

Within the context of this post we are going to focus on a fairly common problem and that is enrolling new customers.  As I previously mentioned, customers can choose their retailer.  Like most businesses retailers are very interested in on-boarding new customers with incentives.  When a customer does choose to move their business to a new retailer, this retailer needs to inform the Distribution provider that the customer has chosen to switch retailers.  The Distribution provider is then responsible for notifying the old retailer that this customer is now longer a customer of theirs and the Distributor is also responsible for confirming with the new retailer that this customer is now theirs.  From here on in, the Distribution company will send this customer’s consumption (Meter Reads) to the new retailer so that they can bill the customer.

This process is defined as a set of transactions known as the Enrollment transactions and is comprised of three messages:

• SRR – Switch Retailer Request
• SRN – Switch Retailer Confirmation for new Retailer
• SRO – Switch Retailer Confirmation for old Retailer

Implementation

All participants within the market place have their own company identifier.  For this blog post I have created fictitious ids.  Our New Retailer will have an id of 1111, our Old Retailer will have an id of 2222 and our Distribution Company will have an id of 3333.These ids will map to Queues that have been created within the Service Bus.

1. The New Retailer will send an SRR request to the Distribution Company’s queue called 3333.
2. The Distribution Company will pull this message, via BizTalk,  off of its queue and communicate with its back end ERP system (not described within this blog post).
3. BizTalk will then need to send a notification (SRN) back to the new Retailer confirming the enrollment.
4. BizTalk will also need to send a notification (SRO) to the Old Retailer letting them know that they have lost a customer.

 

Service Bus Configuration

Within my Azure account I have created a new Namespace called Electricity and created 3 different queues:

• 1111
• 2222
• 3333

.Net Retailer Clients

Similar to some of my other recent posts we are going to use the .Net Brokered Message API from client applications that will interact with BizTalk through the Service Bus.

Within our .Net Solution there are 3 projects:

• DataContracts -  where we will find our message specifications for the SRR, SRN and SRO message types.
• Retailer1111 – where we will find our New Retailer  functionality.  This project will Send the SRR and receive the SRN response.
• Retailer2222 – where we will find our Old Retailer functionality.  This project will Receive the SRO confirmation.

DataContracts

The first class that we want to create represents the Switch Retailer Request (SRR).  Within this message we have properties for SiteID (Customer identifier), New Retailer ID, Old Retailer ID and the Enrollment Date (effective date).

Normally the New Retailer would not necessarily be aware of the Old Retailer ID but since I am leaving the ERP functionality out of the scope of this blog post we will assume that the New Retailer is aware of this information.

namespace DataContracts
{
    public class SwitchRetailerReqeust
    {
        public string SiteID {get;set;}
        public string NewRetailerID {get;set;}
        public string OldRetailerID { get; set; }
        public DateTime EnrollmentDate { get; set; }
    }
}

 

Next we will get into the Switch Retailer Notification (SRN).  This is the message that BizTalk will send to the New Retailer confirming that the customer is now theirs.

namespace DataContracts
{
    public class SwitchNewRetailerNotification
    {
        public string SiteID { get; set; }
        public string NewRetailerID { get; set; }
        public DateTime EnrollmentDate { get; set; }
     
    }
}

 

Finally, we have the Switch Retailer Notification (SRO) that needs to be sent to the Old Retailer letting them know that this customer is no longer theirs.

namespace DataContracts
{
    public class SwitchOldRetailerNotification
    {
        public string SiteID { get; set; }
        public string OldRetailerID { get; set; }
        public DateTime EnrollmentDate { get; set; }
    }
}

Retailer1111 Project

As previously stated, the purpose of this project is to send the SRR transaction to the Distribution company and then receive the corresponding SRN transaction back.

using System.Text;
using System.Threading.Tasks;
using Microsoft.ServiceBus;
using Microsoft.ServiceBus.Messaging;
using DataContracts;
using System.Runtime.Serialization;
using System.IO;

namespace RetailerNew
{
    class Retailer1111
    {
   

            const string SendQueueName = "3333";
            const string ReceiveQueueName = "1111";
            const string ServiceNamespace = "<your_namespace>";
            const  string IssuerName ="owner";
            const string IssuerKey = "<your_key>”;

        static void Main(string[] args)
        {
            //***************************************************************************************************
            //                                   Get Credentials
            //***************************************************************************************************          
            TokenProvider credentials = TokenProvider.CreateSharedSecretTokenProvider  (Retailer1111.IssuerName, Retailer1111.IssuerKey);
            Uri serviceUri = ServiceBusEnvironment.CreateServiceUri("sb", Retailer1111.ServiceNamespace, string.Empty);

            MessagingFactory factory = null;

            try
            {
                //***************************************************************************************************
                //                                   Management Operations
                //***************************************************************************************************          
                NamespaceManager namespaceClient = new NamespaceManager(serviceUri, credentials);
                if (namespaceClient == null)
                {
                    Console.WriteLine("\nUnexpected Error: NamespaceManager is NULL");
                    return;
                }

                Console.WriteLine("\nChecking to see if Queue '{0}' exists...", Retailer1111.SendQueueName);

                // If Queue doesn't exist, then let's create it
                if (!namespaceClient.QueueExists(Retailer1111.SendQueueName))
                {
                    QueueDescription qd = new QueueDescription(Retailer1111.SendQueueName);
                   

                    namespaceClient.CreateQueue(qd);
                  
                }

                //***************************************************************************************************
                //                                   Runtime Operations
                //***************************************************************************************************
                factory = MessagingFactory.Create(serviceUri, credentials);

                QueueClient myQueueClient = factory.CreateQueueClient(Retailer1111.SendQueueName);

                //***************************************************************************************************
                //                                   Sending messages to a Queue
                //***************************************************************************************************
               

                Console.WriteLine("\nSending messages to Queue...");

                //Create a Switch Retailer Request
                SwitchRetailerRequest srr = new SwitchRetailerRequest();
                srr.SiteID = "3333123456789";
                srr.NewRetailerID = "1111";
                srr.OldRetailerID = "2222";
                srr.EnrollmentDate = DateTime.Now.AddDays(1);
          

     //Serialize the request so that BizTalk can process it correctly

                BrokeredMessage message = new BrokeredMessage(srr, new DataContractSerializer(typeof(SwitchRetailerRequest)));

                //Here we specify which URI we are expecting our response
                message.ReplyTo = serviceUri.AbsoluteUri + Retailer1111.ReceiveQueueName;
                 myQueueClient.Send(message);

                 //**************************************************************************************************
                 //                                   Receive messages from Queue
                 //**************************************************************************************************

                TokenProvider tokenProvider = TokenProvider.CreateSharedSecretTokenProvider(
                    Retailer1111.IssuerName, Retailer1111.IssuerKey);
                Uri uri = ServiceBusEnvironment.CreateServiceUri("sb", Retailer1111.ServiceNamespace, string.Empty);
                MessagingFactory messagingFactory = MessagingFactory.Create(uri, tokenProvider);
                QueueClient qc = messagingFactory.CreateQueueClient(Retailer1111.ReceiveQueueName, ReceiveMode.ReceiveAndDelete);
                BrokeredMessage bm;
                while ((bm = qc.Receive(new TimeSpan(hours: 0, minutes: 0, seconds: 30))) != null)
                {
                    var data = bm.GetBody<SwitchNewRetailerNotification>(new DataContractSerializer(typeof(SwitchNewRetailerNotification)));
                    Console.WriteLine(String.Format("Customer with SiteID {0} has now been enrolled as of {1}", data.SiteID, data.EnrollmentDate.ToString() ));
                }

                Console.WriteLine("\nAfter running the entire sample, press ENTER to clean up and exit.");
                Console.ReadLine();
            }
            catch (Exception e)
            {
                Console.WriteLine("Unexpected exception {0}", e.ToString());
                throw;
            }
            finally
            {
                // Closing factory close all entities created from the factory.
                if(factory != null)
                    factory.Close();
            }
           
        }

    }

}

 

Probably the most interesting/significant line of code in there is where we set the Brokered Message ReplyTo property.  The reason why this code is significant is that BizTalk can use the value of this property to set our URI that our Dynamic Send port is going to use in order to send the response back out.  You will see how this is set in the BizTalk section.

//Here we specify which URI we are expecting our response
message.ReplyTo = serviceUri.AbsoluteUri + Retailer1111.ReceiveQueueName;

 

Retailer2222 Project

The purpose of this project is to retrieve the SRO notifications that occur when we lose a customer to another retailer.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using Microsoft.ServiceBus;
using Microsoft.ServiceBus.Messaging;
using DataContracts;
using System.Runtime.Serialization;
using System.IO;

namespace RetailerOld
{
    class Retailer2222
    {

       
        const string ReceiveQueueName = "2222";
        const string ServiceNamespace = "<your_namespace>";
        const  string IssuerName ="owner";
        const string IssuerKey = "<your_key>";

        static void Main(string[] args)
        {

               try
               {

                //Create instance of tokenProvider using our credentials
                TokenProvider tokenProvider = TokenProvider.CreateSharedSecretTokenProvider(
                    Retailer2222.IssuerName, Retailer2222.IssuerKey);
                Uri uri = ServiceBusEnvironment.CreateServiceUri("sb", Retailer2222.ServiceNamespace, string.Empty);
                MessagingFactory messagingFactory = MessagingFactory.Create(uri, tokenProvider);
                QueueClient qc = messagingFactory.CreateQueueClient(Retailer2222.ReceiveQueueName, ReceiveMode.ReceiveAndDelete);
                BrokeredMessage bm;

                //***************************************************************************************************
                //                                   Receive messages from Queue
                //***************************************************************************************************

                Console.WriteLine("About to connect to the Queue");
                while ((bm = qc.Receive(new TimeSpan(hours: 0, minutes: 0, seconds: 30))) != null)
                {
                    var data = bm.GetBody<SwitchOldRetailerNotification>(new DataContractSerializer(typeof(SwitchOldRetailerNotification)));
                    Console.WriteLine(String.Format("Customer with SiteID {0} is no longer our customr as of {1}", data.SiteID, data.EnrollmentDate.ToString() ));
                }

                Console.WriteLine("\nAfter running the entire sample, press ENTER to clean up and exit.");
                Console.ReadLine();
            }
            catch (Exception e)
            {
                Console.WriteLine("Unexpected exception {0}", e.ToString());
                throw;
            }
       
        }
}
}

 

BizTalk Project

Within this solution I am going to demonstrate what is required to send messages to Service Bus Queues using a dynamic send port.  I am going to use two slightly different approaches so keep your eyes open for that.

Schemas

I have created 3 schemas to match those classes that were included in our DataContracts project.  In order to create these schemas, I did use the Auto Generate Schemas feature that is included in Visual Studio (when you have BizTalk installed of course).  I discussed this approach in a previous post if you are interested in more details.

Maps

I have two very simple maps, the first one will transform an instance of our SRR message to an instance of our SRN message.

The second map will transform an instance of our SRR message to an instance of our SRO message.

Orchestration

Here is where most of the “magic” happens.  We have a receive shape where we will receive an instance of our SRR message.  We will then transform it and set some context properties, within a Message Assignment shape, so that we can use a Dynamic Send Port.  What is special about this Message Assignment shape is that we are going to use a Brokered Message property called ReplyTo as our URI.  So this is a pretty need “out of the box” feature that allows a client to dictate where we need to send a response message.

Even thought this ReplyTo property is beneficial, it only gets us part way.  We will need to provide a transport type and indicate that it is the new SB-Messaging adapter.  We also need to provide our Service Bus credentials and the URI to the Access Control Service.  It, of course, is a good idea to maintain these values in some sort of configuration store as opposed to hardcoding within our Orchestration.

Once we have sent out our SRN message back to the Service Bus Queue, we now need to process the SRO message.  We will once again leverage a map to instantiate this message in the absence of an ERP system were this message would ordinarily be created.  We will once again take advantage of Dynamic Send Ports but in this case we do not have the ReplyTo brokered message property because this retailer never sent us the message.  We will need to provide the base uri for the Service Bus but we can augment this by distinguishing the OldRetailerID node that is part of the SRO schema.  This will allow the solution to work for any retailer that needs to receive an SRO.

Much like the previous scenario we will need to specify a TransportType, our credentials and our Access Control Service URI.

When we are done putting these building blocks together we should have something that resembles the following.  We can now deploy our BizTalk application.

Receive Location

We will configure our Receive Location much like we did in my previous posts that discuss BizTalk integration with the Service Bus.

Send Ports 

As outlined in the Orchestration section of this post, we have two different Send Ports.  Part of the reason why I have included two is to demonstrate the following NEW features that has been included in the BizTalk 2013 Beta.  In previous BizTalk versions, if you used a Dynamic Send Port then it would automatically use the Adapter’s Default Send Handler to send out the message.  This isn’t ideal, especially in a shared environment where there can be many different groups or applications using this same Send Handler(Host).

Going forward we now have a Configure button that we can click within a Dynamic Send Port’s configuration.  When we do this a Configure Send Handler dialogue will appear that allows us to set different Send Handlers depending upon the Adapter.

Even within our own Application we can set a different Send Handler if you have more than 1 Dynamic Send Port.  For our second Dynamic Send Port I have specified BizTalkServerApplication as the Send Handler for this Dynamic Send Port.

There is probably not a really good reason to do this but the point I am trying to make is that we have gone from being very restricted to having a lot of flexibility.

Testing

We are now ready to test our application.  The steps that we need to take in order to do so are:

• Run an instance of our Retailer1111 client.  This application will send the SRR to the Service Bus Queue where BizTalk will retrieve it.
• BizTalk in turn will look for the ReplyTo property and send an SRN message back to the ServiceBus but this time to the the Retailer’s 1111 queue.
• Next, BizTalk will generate an SRO message and send it to the old Retailer’s 2222 queue.

Conclusion

The point of this blog post was to take a real world problem and discuss how the combination of Azure Service Bus Queues and BizTalk 2013 can help us solve it.  Of course I had to over-simplify things to make this blog post some-what reasonable in length.  This also is not the only approach that we can take in order to solve this problem.  Using Azure Service Bus Topics is another option as well but I thought it was important to demonstrate how we can use the SB-Messaging context properties and Dynamic Send Ports.  I also felt it was a great opportunity to highlight the new Send Port Configuration for Dynamic Send Ports.