Lubo Blagoev's Blog

Hello everyone. Its been a while since i posted a anything here.

This is what to become a series of posts about how to do a Load Balanced Server Farm using WCF and MSMQ. In this post i will present the problem, then the approach and then I want to show the hiccups you may get trying to do this the right way.

Now i suppose you are familiar with WCF and MSMQ and most probably you are since you googled-binged your way to get here. I am going to omit some of the general details.

Scenario

Your boss said:

“I want to build a load balanced server farm that can load balance long running tasks from here to the moon.”

or what we want to achieve is a server farm represented by many services/agents running on the node machines reading from one single queue. You have clients posting to this queue and you want the agents to read the messages and process them independently from one another. In theory all agents will fight for messages from the queue doing the load balancing we want to achieve. If an agent gets a job done it will go and take another job from the queue. This problem in fact falls perfectly within the Analytic Queuing Theory described in plain words by Eric Lippert’s Queueing Theory In Action, plus, frogs. You do read his blog right? I won’t go into details but its the W system we are trying to build here.

The principle downside of the W system is that the single queue looks like it will take much longer than four short queues in the M system, which can be daunting. But by almost every relevant objective metric, by almost every relevant social factor, and in almost every common real-world business scenario the W system is preferable

Why MSMQ?

This is easy to understand since we get all the goodness for free.  We get security, message persistence, durability, transaction support, retry handling, failover, asynchronous processing (needed when long running jobs right?), timeouts, time to live, scalability (new nodes arriving/leaving) and so on and so forth.

Why WCF?

Is there any other way? To work with remote queues though you will need to fall back to msmqIntegrationBinding. The netMsmqBinding does not work with remote queues in workgroup mode. I did some digging and found that the netMsmqBinding should be able to work with remote queues but the address translation doesn’t translate correctly direct format names. That’s the only problem that stops this binding of opening a remote queue in workgroup mode. That’s a pity!

Another drawback is an existing issue of message pre-read. This is when you read one message to process then WCF reads some more message/messages from the queue. This essentially locks them and no other agent can read them. They call it pre-reading. This harms the farm scenario very much since while one message is processed other is waiting on the same agent when there are other agents available to process it. That should be fixed in .NET 4.0 RTM. In .NET 4.0 beta 2 its fixed for netMsmqBinding only.

Target Environment

The environment affects the choices you need to make the server farm work. Its configuration choices mostly, but it can bubble up to your service and client code as you will see. Lets consider the target environment:

First there is a “Standard” environment. This includes .NET 3.0, 3.5 with/without SP1, and a mixture of Windows 2003/2008/XP/Vista/Win7 machines.

and then there is this “Modern” environment that includes .NET 3.5/4.0 beta 2 (RTM is not released yet) and Vista/Win7/Server 2008 only machines.

Standard: This means you are using MSMQ 3.0 which means no transactional remote read from the queue. Having at least one agent running on a pre-Vista OS breaks everything and you need to rely on specific architecture to support transactions. But there is a solution. You will need to build Transactional Read-Response. That is something i never did so please share your thoughts of how it shapes-up in real world.

Modern with .NET 3.5 requires you to read with transactions from the remote queue to have any kind or durability. These transactions don’t offer the best performance since you need a transaction even when you are not sure you will be able to handle the message. That’s should be fixed in .Net 4.0 with the new ReceiveContext support and the new Alternative Queuing Model

Modern with .NET 4.0 will become standard after some time. Now it is considered modern. It has the all the goodness - remote transacted read, the ReceiveContext support, No message pre-read, custom dead letter queues etc. It requires again the msmqIntegrationBinding since the directly address formatting issue is still present. Lets hope this will be fixed in the RTM.

In the next post i will present a solution of the scenario we are trying to build. I will show how to fix the pre-read issue on msmqIntegrationBinding that may work on  both .NET 3.5 and .NET 4.0.

[Update:] v.1.2 Fixed changing a file attachment to a new file with the same filename

[Update:] v.1.1 Added a tooltip on long filenames

I wrote a simple plugin for Windows Live Writer.

Attach File Plugin allows you

• To attach a file into your blog post.
• Modify attachment properties from WLW Sidebar
• Include Google tracking code for the attached file.
• You can upload the files using FTP if you set WLW to use ftp for images.

One important fact Windows Live Writer handles post files very bad. The API and implementation is awful. Test it for yourself . Add a picture to your post. Note in source the href of that picture. Change a property of that picture. (Size, Borders, Wrappings, Margins). Note the href again. Did it change? Yes it did. What happened is that WLW copy the picture every time you change something. You end up with a file name “Me and my girl at the beach[0][1][2][0][9].jpg”. More over they do this on the UI thread. Yes If I want to insert a 25MB picture it gets copied over and over again on the UI thread. It takes 7 seconds right now to get to a blinking cursor from a selected image on my Dual Core 2,4GHz, 4GB RAM machine. I guess they do some serialization of the post at that time. I noticed they call SmartContentSource.GeneratePublishHtml() many times.

It gets worse as you dig in. Plug-ins are able to kill the hosting process by just accessing publishingContext properties in GeneratePublishHtml method.

You can’t post files with spaces in the name and more than certain limit on the filename length. You save a draft post, restart WLW open the draft and voila you have a href to a file “ConcurencyandParallelProgra.txt” instead of the original “Concurency and Parallel Programming.txt”

The docs for Distributing Plugins are wrong. Do not put your registry settings under “HKEY_CURRENT_USER\SOFTWARE\Windows Live\Writer\PluginAssemblies”, instead use “HKEY_CURRENT_USER\SOFTWARE\Windows Live Writer\PluginAssemblies”. They enumerate both paths but load dlls only from the later.

That’s the reason I wrote this plugin myself. None of the plug-ins that add files to your post handles these issues. Not even the default Microsoft plug-ins. (Insert Picture) I hope i managed to work around these problems.

Lastly Google PageView option allows you to track downloads of the attached file. Refer to this page for more.

Example: If you set the PageView to “my file” the generated html will be

<a href="http://www.example.com/files/map.pdf" onClick="javascript: pageTracker._trackPageview('my file'); ">

Here are the installation packages and source code

P.S. I will post the plugin to Windows Live Writer Gallery later since the site is not working right now. (ARGHHHHHH!!!)

Lubo.

Hello everybody. I bought a car not so long ago. It was in October 2008 but just now I found time to write about it.

It’s an Audi A4 163 hp year 2004. I got it in perfect condition at around 46,000km. The previous owner didn’t used it a lot since he had many Audis in the garage.

Yeah! What a car!

Now you know why it took me so long to pay attention to my blog

I got some manuals with the car so I decided to share them with you. I did some scanning , cropping, merging for a couple of days and here they are:

Enjoy reading. I am going for a ride! (…speed… give me what I need… yeaaaah!!!)

Do you want go to the moon? You think it's not possible. Well it is. NASA collects names to put onboard the Lunar Reconnaissance Orbiter. You can't go exactly but you can show your support to the program by participating. Just enter your first and second names on the Name To The Moon form its as simple as that.

LRO's objectives are to find safe landing sites, locate potential resources, characterize the radiation environment, and demonstrate new technology. If you want to know more visit LRO's home page at http://lunar.gsfc.nasa.gov/

Here's the certificate I got

If you think layered application design is good and it's the best way to throw some work at the juniors in you team think again.

If you want to learn more about

• Inner Dependency Problem
• Dependency Inversion
• Service locator
• Poor's man Dependency Injection
• Inversion of Control Containers
• Static Gateway Pattern
• What's wrong with singletons
• Boo programming language
• Decorator pattern
• Coding to contract rather than concrete implementation
• Dependency Resolver

there is a very good MSDN article about software component dependencies. I suggest you read it if you haven't already.

Tame Your Software Dependencies for More Flexible Apps (By James Kovacs)

Unfortunately the author missed to mention about Object Builder nor the recently announced Unity Application Block but I think you can investigate further on your own.

Loosen up!
Lubo.