Multiple MessageBox databases in a BizTalk Server group

My current client is one of the retail giants in UK. During the Christmas period, their numbers of business transactions are always very heavy. So, all of their IT applications will face heavy load during October - December period. As part of preparing for that period, the peak readiness activity will start in their initial part of the business year which starts from February. I have been employed to prepare their integration services applications to handle their projected high business transactions during Christmas. Many ideas were discussed for the integration applications which uses BizTalk server. One of them is, empowering the existing BizTalk servers inventory with more servers for scalability and reliability. The scalability of the BizTalk Server databases can be achieved by adding multiple Messagebox databases. The reliability for the BizTalk Server databases can be achieved by using clustered server mechanism. In this post, I am going to discuss about using multiple Messagebox databases in a BizTalk Server group.

First, we will see when we need to use multiple MessageBox databases.

We use the multiple MessageBox databases, when we have

• More database transactions which increases the Messagebox database size and results in lesser performance. For example, when the number of messages to the MessageBox is more.
• Heavy message interactions which is database-intensive and results in lesser performance. For example, if the messages interact with the MessageBox are heavy in size.

The MessageBox database performs three main functionalities. They are:

• Evaluate subscriptions.
• Message publication.
• Save the message state.

So when we decide to add multiple MessageBox databases, our intention should be to distribute the functionalities to multiple databases/servers rather than sharing these functionalities with multiple databases/servers. We will see more detail of this in the following section.

When we add the first MessageBox database to the BizTalk processing servers using the Configuration Wizard, we need to make this MessageBox database as Master MessageBox database. Master MessageBox database is one of the MessageBox databases in your BizTalk Server group which holds the master subscription information, performs the subscription evaluation functionality of the MessageBox and routes the messages to the appropriate MessageBox databases for active message publication. Yes, Master MessageBox database doesn't perform the message publication. It only performs the "Evaluate subscriptions". This is done by disabling the new message publication in the Master MessageBox database.

Figure 1: Disabling new message publication for Master MessageBox Database.

A Master Messagebox database can receive two types of messages:

• A new activation message.
• An active instance's message/Correlation message.

Master MessageBox database with new activation message:

When the new activation message received by the Master MessageBox, after evaluating the subscription the Master MessageBox database distributes the activation message to the next available MessageBox database(secondary). When you have more than one secondary MessageBox database, Master MessageBox database uses it own build-in logic for load balancing the message for the publication with secondary MessageBox database.

Note: Secondary MessageBox databases are the MessageBox databases which are other than Master MessageBox database in your BizTalk Server group. Though "Secondary" is not the official term, it is used to refer the non-Master MessageBox databases in the group.

Master MessageBox database with Correlation message:

When the correlation message is received by Master MessageBox database, it performs the lookup operation in the database for the secondary MessageBox databases that contains the state for the correlation message. And it distributes the message instance to the secondary Messagebox database which is waiting for this message. In this case, the Master MessageBox database doesn't perform round-robin pattern for message distribution. Rather it distributes to the specific secondary MessageBox database which is already expecting for this message from Master MessageBox database.

During dehydration, the Orchestration engine stores the in-memory representation of the Orchestration, its associated messages and state in the secondary MessageBox database which activated its instance.

Ok, we have decided to add multiple MessageBox databases in the BizTalk Server group. But how many is "multiple"? A minimum of 3 MessageBox databases (as recommended by Microsoft) for a BizTalk Server group. Out of these one should act as the Master MessageBox database (needs to disable the message publication on this MessageBox database) and other two should act as the secondary MessageBox databases. This recommendation is made because additional MessageBox databases incurs overhead by the Master MessageBox database for routing between the MessageBox databases. If you have only two MessageBox databases for a BizTalk Server group then most of the benefits gained by the additional MessageBox database is offset by the overhead consumed by the Master MessageBox for message routing. You have another option of using 2 MessageBox databases both performing the subscription evaluation and message publication with your own load balancing logic. In this case distributed transactional consistent with the MessageBoxes cannot be achieved, and hence this method should not be implemented.

So it is always

Master MessageBox database + n- Secondary MessageBox databases.

Another colleague of mine asked can we have multiple MessageBox databases in the same server. Yes, we can have this, but ensure that the existing database server has enough CPU and I/O resources and is bottlenecked only by the lock contention. In most of the enterprises situation this may not be the case. So by adding the secondary MessageBox database on the same physical server can cause server-intensive problems rather than solving the overhead problems involved in the high and heavy message transactions with the MessageBox database server. So it’s always advisable and better to use Messagebox databases in a separate server. For reliability use both the Master and Secondary MessageBox databases in the Clustered server environment.

Since we have started our peak readiness activity much before the Christmas peak time, we had a sufficient time to test the new BizTalk Server group with multiple MessageBox databases in terms of performance and scalability. So thorough testing of your updated server infrastructure is vital in achieving the desired result.

BizTalk Server 2006 - Performance tuning

Kelvin has shared his excellent collection of resources on performance tuning for BizTalk server 2006.

I have started to read through his entire resources. This shall be useful to many other like me.

Thanks Kelvi for sharing your collection.

http://intltechventures.blogspot.com/2008/11/2008-11-01-saturday-biztalk-2006-r2.html

BizTalk testing - Reference

Recently I have read a series of blogs on "BizTalk Testing Guidance".  It details the testing strategies and contains sample test harness code for schema, map, pipleline component, orchestration and .NET component in BizTalk Server. I felt it is interesting and useful, so I thought to refer it.

http://geekswithblogs.net/michaelstephenson/archive/2008/04/27/121687.aspx

Remove a BizTalk server from BizTalk server 2004 group

You must have SQL 2000 Server system administrator rights or BizTalk Server 2004 administrator rights to follow these steps. These steps would only be performed on the BizTalk Processing server has failed as is being replaced.

1. Start the BizTalk Server 2004 Administration snap-in.
2. Expand Servers.
3. Click the computer that is not working.
A list of the host instances that are associated with the computer that is not working appears in the results pane.
4. Right-click each host instance and then click Delete. Repeat this step until all the host instances that are associated with the computer are deleted.

Note: You receive the following error message when you try to delete a host instance that is isolated;

Failed while deleting the Windows NT service BTSSvc {F46166EC-B4B8-4129-AB86-F3BB212CEC50}

You receive the following error message when you try to delete a host instance that is in process;

The network path was not found

If you receive one of these error messages click OK. After you click OK you receive the following error message;

You can forcefully delete this host instance. This may leave orphan orchestrations and prevent cleanup of COM+ applications and NT services on machine ‘computer name’. Do you want to proceed with forceful host instance deletion?

If you receive this error message click Yes.
5. Click Start and then click Run and type wbemtest.exe and then click OK. Hope wbemtest.exe should have already installed in your server.
6. In the Windows Management Instrumentation Tester dialog box click Connect.
7. In the Namespace text box type root\microsoftbiztalkserver and click Connect and the click Enum Instances.
8. In the Class Info dialog box type MSBTS_ServerSetting under Enter superclass name and click to select Immediate only and then click OK.
9. In the Query Result dialog box click the computer that is not working and then click Delete.

Note: You cannot delete the server instance if the server is associated with any host instances.

MoM - Consolidation and Timer based rule

I have a requirement to create a MoM rule which would automate a task based on some specific eventlog entires in production servers. The automatic task is to recycle some of the COM+ components based on the eventlog. So the eventlog entry is the evidence for me to identify that the issue has occurred. I am going not going to talk about the scripts which I have used to restart the COM+ componet but how I have configured the MoM rule for the same.

So I have to create a MoM rule which would monitor the production server and perform the resolution activity. But the crux is to perform the resolution activity only when the continuous 5 entries of the eventlog are identified in the span of 30 seconds in the production server. So I have to create MoM rule which should act based on the number of the entries of an event in the given time window.

I have achieved this by creating an event-rule which would monitor the specified eventlog for the “Repeat Count” of 5 and I have created another consolidation rule which would maintain a counter for event-rule in 30 seconds. This counter will be reset to zero, if the specified count of logs doesn’t met in specified period (30 seconds). i.e. if the application logs 4 event entries for the specified criteria in 30 seconds. The event-rule’s repeat count would be set back to 1. But if the application logs event entry for 5 times in 30 seconds (or in lesser than 30 seconds), the event-rule’s response entries would trigger.

Following are the snapshot of the MoM rules:

Figure 1: Event-rule for the event entry of 5 entries.

Figure 2: Consolidation rule for 30 seconds.

MoM (Microsoft Operations Manager) Design

We have been using MoM (Microsoft Operations Manager) in our BizTalk projects for operational support purposes. I am already very much impressed with the advantages it provides for any Wintel applications in its production support arena. Few months back I got an opportunity to design another MoM package for a messaging product based out of Java for Windows environments. The messaging product has been in use for nearly a year in our customer’s production environment.

When I mean by messaging product, we can’t not compare its capabilities with BizTalk, as it doesn’t perform any business processing upon the messages while transferring the messages. But its intended objective is to just transfer very large files in very high frequency to different platform like UNIX, Mainframe etc. This product is yet to evolve in many ways for an enterprise usage as we all know no application is complete in this universe.

Our customer was already using an in-house operational tool to handle transfer failures (and also to handle the products open issues). This product has very minimal capability in throwing/publishing the exceptions (file transfer failures). Any transfer failures from this product would be simply logged in the event log with same Event ID, Event Category, Event Source and the only difference is the description of the error. Our customer is using an alert monitor tool, which provides the dashboard sort of view to the 1^st line support. So, all the operation tools should provide their output to dashboard tool. They call that tool as Mangers of Manger.

Exceptions in this messaging scenario can be like:

· File not found at source.

· Configured pattern of files are not available for transfer.

· Configured file path / server unavailability.

· File has been occupied by another process

· Server disk space full

· Staging folders maintenance issues

· Many product related issue

· Unknown.

Currently used operational tool doesn’t have the intelligence to classify the exception and act accordingly/show the classified alerts to the Monitoring tool. So the challenge is to create an efficient MoM-package which would identify the different categories of the alerts and give the output accordingly to the Managers of manager tool.

I have classified the exception as below:

The exceptions are classified into different category since each category requires different actions:

• Subscriber issues require Call-out to the Subscriber production-support team.
• Messaging service issue requires Call-out to our production team.
• Product issue requires some workarounds which can be automated again using MoM.
• Off course we can’t always capture all the exceptions, so a generic catch to capture these exceptions also and throw them with lower severity to the monitoring tool.

In MoM with its ability to use custom scripts, I have configured the categories of the exceptions, by throwing them with Custom Exception Number and Custom event description lets say exception category appended at the starting of the every event description. So I have configured a generic alert called “Event-Filter” which swallows all events from the application. In the event-filter I have used a script which would read the event description and in turn throws another alert with different event ID(specific to category) and the event category append in the event description. For example: if the exception is “File not found”

The messaging product would throw the exception as following in the event log:

Event Type : Error Event Source : ABCD Event Category : None Event ID : 123 (Always same for all error) Date : 28/04/2008 Time : 15:21:48 User : Computer : DummyMach123 Description : Files not found at the configured path \\mshsrmsappp0008\P1102747\TestFile.txt>

This event log would be swallowed by the Event-Filter script to throw a MoM event:

Event-ID: 1001 (Custom defined) Event Category: Subscriber Event-Decription: Event_Category [Subscriber]: Files not found at the configured path file:////mshsrmsappp0008/P1102747/TestFile.txt

And for the alerts which came out of the event-filter I have configured a rule to action the category related actions. So the new MoM-package would enable the application to behave something like the following:

Following is the part of the event-filter script which is in if-condition, checks the event description and calls another function (CreateEvent) to throw the event with different event criteria’s.

If InStr(strParentEvtDesc,strSCNoMatchPattern) Then 'match the pattern strParentEvtDesc= strECSubscriber & objContextEvent.Message CreateEvent EVENT_ID_NoMatchPattern,EVENT_TYPE_ERROR,SCRIPT_EVENTSOURCE,strParentEvtDesc Elseif…

Following is the method used to throw the event from event-filter:

Sub CreateEvent(intEventNumber,intEventType,strEventSource,strEventMessage) Set objEvent = ScriptContext.CreateEvent() objEvent.EventSource = strEventSource objEvent.EventNumber = intEventNumber objEvent.EventType = intEventType objEvent.Message = strEventMessage ScriptContext.Submit objEvent End Sub

Following is the event rule in MoM for the subscriber related issue and marked is the criteria for the rule:

As you would have noticed, in the new-MoM package design diagram (showed above) I have showed a XML- file next to the subscriber issues box. Okay I am able to classify the exceptions and now what I am going to do with it? If it’s a subscriber related issue, as I mentioned earlier we want throw a MoM alter which would contain the details of the specific subscriber. I am going to maintain the subscriber contacts in the XML file which can be used a configuration file for the alert for my subscriber related issues. How I am going to accomplish this requirement? We will see it in another day.

Special Thanks to Mr.Lee Nichol for whatsoever little knowledge I have in MoM