Lubo Blagoev's Blog

Hello everyone. Its been a while since i posted a anything here.

This is what to become a series of posts about how to do a Load Balanced Server Farm using WCF and MSMQ. In this post i will present the problem, then the approach and then I want to show the hiccups you may get trying to do this the right way.

Now i suppose you are familiar with WCF and MSMQ and most probably you are since you googled-binged your way to get here. I am going to omit some of the general details.

Scenario

Your boss said:

“I want to build a load balanced server farm that can load balance long running tasks from here to the moon.”

or what we want to achieve is a server farm represented by many services/agents running on the node machines reading from one single queue. You have clients posting to this queue and you want the agents to read the messages and process them independently from one another. In theory all agents will fight for messages from the queue doing the load balancing we want to achieve. If an agent gets a job done it will go and take another job from the queue. This problem in fact falls perfectly within the Analytic Queuing Theory described in plain words by Eric Lippert’s Queueing Theory In Action, plus, frogs. You do read his blog right? I won’t go into details but its the W system we are trying to build here.

The principle downside of the W system is that the single queue looks like it will take much longer than four short queues in the M system, which can be daunting. But by almost every relevant objective metric, by almost every relevant social factor, and in almost every common real-world business scenario the W system is preferable

Why MSMQ?

This is easy to understand since we get all the goodness for free.  We get security, message persistence, durability, transaction support, retry handling, failover, asynchronous processing (needed when long running jobs right?), timeouts, time to live, scalability (new nodes arriving/leaving) and so on and so forth.

Why WCF?

Is there any other way? To work with remote queues though you will need to fall back to msmqIntegrationBinding. The netMsmqBinding does not work with remote queues in workgroup mode. I did some digging and found that the netMsmqBinding should be able to work with remote queues but the address translation doesn’t translate correctly direct format names. That’s the only problem that stops this binding of opening a remote queue in workgroup mode. That’s a pity!

Another drawback is an existing issue of message pre-read. This is when you read one message to process then WCF reads some more message/messages from the queue. This essentially locks them and no other agent can read them. They call it pre-reading. This harms the farm scenario very much since while one message is processed other is waiting on the same agent when there are other agents available to process it. That should be fixed in .NET 4.0 RTM. In .NET 4.0 beta 2 its fixed for netMsmqBinding only.

Target Environment

The environment affects the choices you need to make the server farm work. Its configuration choices mostly, but it can bubble up to your service and client code as you will see. Lets consider the target environment:

First there is a “Standard” environment. This includes .NET 3.0, 3.5 with/without SP1, and a mixture of Windows 2003/2008/XP/Vista/Win7 machines.

and then there is this “Modern” environment that includes .NET 3.5/4.0 beta 2 (RTM is not released yet) and Vista/Win7/Server 2008 only machines.

Standard: This means you are using MSMQ 3.0 which means no transactional remote read from the queue. Having at least one agent running on a pre-Vista OS breaks everything and you need to rely on specific architecture to support transactions. But there is a solution. You will need to build Transactional Read-Response. That is something i never did so please share your thoughts of how it shapes-up in real world.

Modern with .NET 3.5 requires you to read with transactions from the remote queue to have any kind or durability. These transactions don’t offer the best performance since you need a transaction even when you are not sure you will be able to handle the message. That’s should be fixed in .Net 4.0 with the new ReceiveContext support and the new Alternative Queuing Model

Modern with .NET 4.0 will become standard after some time. Now it is considered modern. It has the all the goodness - remote transacted read, the ReceiveContext support, No message pre-read, custom dead letter queues etc. It requires again the msmqIntegrationBinding since the directly address formatting issue is still present. Lets hope this will be fixed in the RTM.

In the next post i will present a solution of the scenario we are trying to build. I will show how to fix the pre-read issue on msmqIntegrationBinding that may work on  both .NET 3.5 and .NET 4.0.