Using Wireshark to capture the SOAP messages

Again this blog is my knowledge repository/diary. I have done something new which I haven’t done before; I have blog it here so that I can refer it when I want to reuse it again.

I was working with a webservice to which I would post a request message from my test harness and expected the response / resultset from it. But the response was an unexpected resultset. So I wanted to capture the outgoing request to the webservice and incoming response from it by intercepting and examining the SOAP messages exchanged to and from the webservice.

Before I was using an excellent tool to monitor traffic over a machine NIC called Packet sniffer. Now I wanted to use Wireshark (http://www.wireshark.org/download.html) another excellent network sniffing tool.

I have installed and running the Wireshark on my test machine where I am going to post the request message using the test harness to the webservice. Off course you can use Wireshark on any end, either at test harness end or at Webservice end. This is just a network sniffer, which can sniff both the incoming and outgoing messages.

Click “Capture” and select “Options”

Below is the screen shot of “Wireshark:Capture Options”.

Then do the following steps:

Step 1: In “Interface” dropdown list, select the appropriate network card to capture data from. There will be only one option if you have have only one ethernet card. If you’re not sure, no worries try by selecting the different network card and see whether the tool captures any data for the selected card. If it doesn’t, then change the option.

Step 2: Enter “tcp port 8080” in the “Capture Filter:” This filter instructs the Wireshark to capture TCP packets over the port 8080. You can get the port number from the webservice object’s URL property or from the Endpoint definition of your webservice.

Step 3: Click “Start”

Now Wireshark is ready to capture the network traffic over the port 8080. Send some calls to the webservice. After sending the request and receving the response, stop the capture by selecting Capture->Stop or press Control+E.

By default you will have three panel view: Packet List, Packet Details and Packet Bytes.

Packet List is at the top, Packet Details is in the middle and PacketBytes is at the bottom of the window.

Now you have captured the all the network traffics over  the port 8080. All of them would be listed in the Packet List panel. Select one by one and see their details in the Packet Details panel. In the Packet Detail panel you will have tree node like structure. Expand the Hypertext Transfer Protocol, and then look for a node called “Data:” under it. Select the node “Data:” and in the Packet Bytes panel you would see the actual data captured, as shown in the above picture. To read the data in a better way, right-click on the “Data:” node in the Packet Details panel, select Copy->Bytes (Printable Text only) and paste it in a note pad to see the exact xml message being passed to the webservice.

<?xml version="1.0" encoding="utf-8"?>

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">

<soap:Body>………………………………………………..

This is the SOAP request sent across to the webservice and we will capture the SOAP response from the webservice. In the Packet List panel, go through the list after the SOAP request. And you will see the SOAP response on the Packet Details panel itself.

VBScript to convert the comma-delimited file to a carriage-return delimited file

'This script will convert the comma-delimited file to a carriage-return delimited file.

Const ForReading = 1
Const ForWriting = 2

Set objFSO = CreateObject("Scripting.FileSystemObject")
Set objFSOWrite = CreateObject("Scripting.FileSystemObject")

'Set folder
objStartFolder = "C:\Folder"

Set objFolder = objFSO.GetFolder(objStartFolder)

Set colFiles = objFolder.Files

'Loop through all the files in the folder
For Each objFile in colFiles
    'This line will output the name of the file including the extension
    objFileNameParts = split(objFile.Name,".")
    'Remove the extension from the file name
    objFileName = objFileNameParts(0)   
    'msgbox("FileName: " & objFileName)
    objReadFilePath = objFolder & "\" & objFileName & ".csv"
    objWriteFilePath = objFolder & "_OUTPUT\" & objFileName & ".csv"
    'msgbox("ReadFilePath: "& objReadFilePath)   
    'msgbox("WriteFilePath: "& objWriteFilePath)
    Set objFile = objFSO.OpenTextFile(objReadFilePath, ForReading)   
    Do Until objFile.AtEndOfStream
        strLine = objFile.ReadLine
        'Replace commas(,) with carriage return(vbCrLF)
        strLine = Replace(strLine, ",", vbCrLF)       
        strNewText = strLine   
    Loop
    objFile.Close
    'Create a new file in not created before, otherwise rewrite in the existing file
    If objFSO.FileExists(objWriteFilePath) Then
        Set objFile = objFSO.OpenTextFile(objWriteFilePath, ForWriting)
        MSGBOX("FILE OPENED")
    Else
        set objFile = objFSO.CreateTextFile(objWriteFilePath)
        MSGBOX("FILE CREATED")
    End If
    objFile.WriteLine strNewText

    objFile.Close
Next

KB: The following error occurred while saving message from the message box tracking spool. Class not registered

Today when I tried to save the tracked message in HAT, I had the following error:

The following error occurred while saving message from the message box tracking spool. Class not registered

The solution which worked for us is to restart the “Windows Management Instrumentation” service in the services tray.

HAT  uses WMI to query the MessageBox database. There can be two possible causes for this issue. Connectivity issue to the database or WMI. Since the message transmission is normal and its only the problem with HAT, we didn’t doubt the connectivity. Anyway to ensure that the things are right, we checked the MS-DTC setting as suggested here..http://msdn.microsoft.com/en-us/library/aa544733(CS.70).aspx .

Then WMI, in our case we have HP-Openview (like MOM/SCOM) to monitor the BizTalk environment and its agent service runs in BizTalk server to check the status of the server. We have some rules in Openview to monitor the status of the BizTalk artefacts like Orchestrations, receive locations , send ports etc. which uses WMI. Since Openview continuously polls for the BizTalk servers for the status the WMI services is battered. So for us recycling the WMI service (Windows Management Instrumentation) resolved the issue.

Errors List

In our day to day work, if we face any errors we either resolve the error by our experience or google it to find the correct solution. When you search for the correct solution often we find various potential solutions. So we go through each of them and apply the most suitable for our situation. Why not list the errors which I have encounters here. Instead of searching for the right solution why not first look here and see whether these solution which I have used sometime earlier when I had errors. This may not be useful to others, but might help me again in saving some time. So this is going to be a running list, I’ll update as when I encounter any errors.

Error	The following error occurred while saving message from the message box tracking spool. Class not registered
Cause	WMI service has been battered
Solution	Recycle the “Windows Management Instrumentation” service in the services tray
Reference	http://indway-is.blogspot.com/2010/10/kb-following-error-occurred-while.html

 

Error	Cannot load source/destination schema: “schema_name”. Either the file/type does not exist, or if a project dependency exists, the dependent project is not built
Cause	For some reason the BizTalk designer loses the schema reference and it is a little hard trying to “trick” the designer into remembering the schema’s location
Solution	Unload/load project of refresh the VS shell with the references.
Reference	http://sandroaspbiztalkblog.wordpress.com/2010/04/26/biztalk-mapper-cannot-load-sourcedestination-schema/

Error	Error <<Error Number>> Exception Caught: Mapper Compiler: Functoid not found: guid (<<GUID>>) with functoid id (<<functiod ID>>)
Cause	For some reason the BizTalk designer loses the schema reference and it is a little hard trying to “trick” the designer into remembering the schema’s location
Solution	Copy the custom functiod’s assembly to Drive:\Program Files\Microsoft BizTalk Server 2006\Developer Tools\Mapper And GAC the custom functiod’s assembly
Reference	http://social.msdn.microsoft.com/Forums/en-US/biztalkgeneral/thread/f9d43d64-84a3-4116-95cb-f4cca19a7d82

Error	sgen.exe exited with code 1
Cause	When you build the solution in release mode. Exact cause??
Solution	Remove the previous version of the assembly from the GAC
Reference	http://chrissyblanco.blogspot.com/2006/10/sgenexe-exited-with-code-1.html

BizTalk Server 2010 - RTM

BizTalk Server 2010 has been released to manufacturing (RTM)and available for purchase from October 1^st, 2010.

This seventh major release of Microsoft’s enterprise integration product’s developer edition is free. So you can start develop BizTalk Server 2010 applications if you have Visual Studio and SQl Server licenses.

Details about its new features: http://blogs.msdn.com/b/biztalk_server_team_blog/archive/2010/09/22/biztalk-server-2010-released-for-manufacturing.aspx

The evaluation and free developer editions are now available for download from: http://www.microsoft.com/biztalk/en/us/blogs.aspx

Which to use: BizTalk or SQL-Integration Service (SSIS) ?

I had a situation where I could have used any of these tools, but after some reading about the difference I have nailed down to use SSIS. So this blog is just a collection of my readings which helped me to differentiate and choose the product in my project.

BizTalk and SSIS are different tools and fit different scenarios.

SSIS is designed to move and manipulate very large amount of data over extremely high performance batch processing where BizTalk has been designed to move, process, validate, transform, and route low rate of transactions consisting of tiny amounts of business data

Use SSIS for:

Large ETL processes like de-batching the large CSV file into SQL database, or cleansing the large record before inserting in to database.

Batch Oriented Scenarios i.e. when the files need to be processed in specific time of the day as opposed to file needs to be processed as and when the files are available.

Use BizTalk:

When you have to do the Business Process Management (BPM). i.e. some business processes.

Near real-time scenarios i.e. as and when the files are available and it has to be processed.

SSIS and BizTalk can work together:

Though SSIS and BizTalk are used in different place depends on the scenario, it’s important to note that we can use SSIS with BizTalk. They can work together. For Example we have a project that would receive a batch of data once a day - containing 1000s of records. SSIS would receive the data, and then scrub the data (or validate the format). Once the data was acceptable, BizTalk would pick up each record and do the business processing.

BizTalk can do SQL-BULK insert but..

For bulk data processing we can use “SQL Bulk Load adapter ” available from http://www.biztalkgurus.com/blogs/biztalksyn/archive/2005/10/23/Processing-a-Large-Flat-File-Message-with-BizTalk-and-the-SqlBulkInsert-Adapter.aspx which can efficiently insert a large amount of data into SQL you are still stuck with the issues of transmitting the MessageBox database and the memory issues of dealing with really large messages.

Reference:

http://social.msdn.microsoft.com/forums/en-US/biztalkgeneral/thread/745b3b31-a83a-4629-bec2-49800e266c7e/

http://geekswithblogs.net/wmichel/archive/2007/04/11/111470.aspx

http://social.msdn.microsoft.com/forums/en-US/biztalkgeneral/thread/745b3b31-a83a-4629-bec2-49800e266c7e/

http://social.msdn.microsoft.com/forums/en-us/biztalkgeneral/thread/261D0FBC-EE91-4211-B811-F1D29A4A38D4

http://social.msdn.microsoft.com/Forums/en-GB/biztalkgeneral/thread/0338b17e-f9af-476c-984f-4982b65f230f

Installed BizTalk Server 2006 R2 – Service Pack 1 (SP1)

Installed the SP1 in our test server yesterday. It was pretty much a smooth ride with minor hurdles. I started off without reading the installation notes (as it’s in *.docx version and we still need to upgrade our MS-Word to latest version. We never felt the need of upgrading the MS-Word until now)

And these error messages didn’t provide much information to resolve. And same as the 13.6 MB log file.

Since our servers are not maintained properly in the recent time with regular reboots, I just rebooted the server and followed the instructions in the installation guide this time (Thought I would perform the installation properly this time, though the document doesn’t provide any clues for the above error).

I’ll emphasis on the followings to be considered before starting the installation:

• SP1-Updates the database schemas:
  • All BizTalk servers in a server farm or group should be updated with the service pack as this updates the database schemas.
  • Backing-up the BizTalk databases are important (as it’s the case for every service pack). If you want to uninstall the SP1 later for any reason, you must restore the BizTalk Server databases with your backup.
• I would always suggest to backup the master secret file.
• All BizTalk Server 2006 R2 hotfixes that are currently installed on the computer will be automatically removed by the service pack installation. Some of the hotfixes are part of the service pack, where some are not. So keep a note of all the BizTalk hotfixes in your server before the installation and compare it against the hotfixes which are part of this service pack(http://support.microsoft.com/kb/974563). If any of your hotfixes are not part of this service pack, you can install the SP1-compatible version later..atleast your note will help you to double check later.

  

• Installing BizTalk Server 2006 R2 SP1 does not change the version number of your BizTalk Server installation. Ours is still “3.6.1404.0” after the service pack installation. (the service packs of previous versions of BizTalk server updates the product version number) So to identify its existence later, Go to “Add or Remove Programs” check the “Show updates” check box and look under “Microsoft BizTalk Server 2006 <<Developer Edition>>”