Adventures with the HTTP Adapter and Yahoo Finance API

I was asked to investigate what would be involved in connecting BizTalk to a Yahoo Finance "api" in order to retrieve stock quotes. This is not a mission critical application but they wanted to be able to consume this information. The client "COTS" application can consume a WSDL, but not a HTTP response that includes comma delmitted data. So we figured that we could expose this data via a web service. There are certainly many ways to expose this data and this is not the point of this post.

The point of the post is to discuss some of the pitfalls that I ran into when trying to connect to this Yahoo API using the BizTalk HTTP Adapter. At first, I thought the problem was rather trivial, I opened IE, pointed it at the Yahoo URL, included the Stock Ticker and the format that I was interested in. The browser returned a string of data that included the stock quote and the other relevant data.

http://download.finance.yahoo.com/d/quotes.csv?s=MSFT&d=t&f=sl1d1t1c1ohgvj1pp2wern


I then saved a copy of this data into a text file, ran it through the BizTalk Flat file schema wizard, created a Receive Pipeline based upon this schema and now had "typed" data for use inside of BizTalk.

At this point, I was a little unsure of what Yahoo was expecting when I made this Http Request. I created a "dummy" schema which only had a root node and figured that I would submit the request to Yahoo to see what was going to happen. Initially, I had a static send port where I hard coded the URL. The URL was very important since it contains the HTTP Request query parameters. I figured that once I get this working, then I will focus on making it dynamic so that the client application can drive which stock quote is returned.

However, this is when I started to run into issues. When I tried to run my application using this configuration I was prompted with the following response from YAHOO: Missing Symbols List.

Based upon this error, I figured that something was up with the query parameters. Yahoo is expecting something along the lines of ?s=MSFT&d=t&f=sl1d1t1c1ohgvj1pp2wern to be past as part of the HTTP Request. After performing some BING searches someone suggested using a dynamic send port to pass these query parameters in. That didn't help either.

I then decided to open up Fiddler to see what was being passed as a successful request. Fiddler is a tool that can be used to inspect HTTP Requests and Responses.
When you use the Request Builder feature in Fiddler, it will default the HTTP request mode to "GET". It makes sense, but I then thought what if I switch this to POST? Look for the red line towards the bottom of the next image: "Missing Symbols List".

At this point, I am starting to understand the problem a little better. After another Bing search, I found the following document that indicates: "The HTTP send adapter gets messages from BizTalk Server and sends them to a destination URL on an HTTP POST request". Using Fiddler, I was able to determine that using a GET request worked without issue. Now knowing that the BizTalk HTTP Adapter is going to use a POST request, I figured that I needed to be able to get Fiddler to work with a POST request and then get BizTalk to use this same approach when posting data to Yahoo.

I am not going to get into the differences between POST and GET here as it has been done so many times before, but here is a good summary of the differences.

Since the query string is essentially being ignored anyways, I removed it from the URL Address text box. I then copied the query parameters into the "Request Body" text box without the '?'

Success! The next challenge is to get BizTalk to pass this data through the HTTP adapter as a POST request.

I found an old forum post by CranCran77 that discussed sending a message of type RawString to a different website that was also looking for HTTP Get. I have used the RawString class before when sending emails via BizTalk, so I was able to add this class to my project quickly.

In the image above, I have highlighted the "Construct Yahoo Request" Message Assignment shape. Below, I have the details of what is inside this message construct shape. Here I am assigning values to my message body that is of type "RawString". This RawString class has been added to a .Net Helper Assembly.

After the message is sent, I can look in the tracking database and can see that this data was transmitted as part of the message body.

Since the parameters that Yahoo requires are being sent as part of the message body, we may use a static Send Port and do not have to provide a query string.

With the emergence of SOAP and now WCF, the use of the HTTP adapter is limited. But as you can see there are still some "services" that exist on the web that rely upon HTTP. Unfortunately there is not a lot of good documentation on the HTTP adapter so hopefully this post fills in some of the gaps.