Lubo Blagoev's Blog

Hello everyone. Its been a while since i posted a anything here.

This is what to become a series of posts about how to do a Load Balanced Server Farm using WCF and MSMQ. In this post i will present the problem, then the approach and then I want to show the hiccups you may get trying to do this the right way.

Now i suppose you are familiar with WCF and MSMQ and most probably you are since you googled-binged your way to get here. I am going to omit some of the general details.

Scenario

Your boss said:

“I want to build a load balanced server farm that can load balance long running tasks from here to the moon.”

or what we want to achieve is a server farm represented by many services/agents running on the node machines reading from one single queue. You have clients posting to this queue and you want the agents to read the messages and process them independently from one another. In theory all agents will fight for messages from the queue doing the load balancing we want to achieve. If an agent gets a job done it will go and take another job from the queue. This problem in fact falls perfectly within the Analytic Queuing Theory described in plain words by Eric Lippert’s Queueing Theory In Action, plus, frogs. You do read his blog right? I won’t go into details but its the W system we are trying to build here.

The principle downside of the W system is that the single queue looks like it will take much longer than four short queues in the M system, which can be daunting. But by almost every relevant objective metric, by almost every relevant social factor, and in almost every common real-world business scenario the W system is preferable

Why MSMQ?

This is easy to understand since we get all the goodness for free.  We get security, message persistence, durability, transaction support, retry handling, failover, asynchronous processing (needed when long running jobs right?), timeouts, time to live, scalability (new nodes arriving/leaving) and so on and so forth.

Why WCF?

Is there any other way? To work with remote queues though you will need to fall back to msmqIntegrationBinding. The netMsmqBinding does not work with remote queues in workgroup mode. I did some digging and found that the netMsmqBinding should be able to work with remote queues but the address translation doesn’t translate correctly direct format names. That’s the only problem that stops this binding of opening a remote queue in workgroup mode. That’s a pity!

Another drawback is an existing issue of message pre-read. This is when you read one message to process then WCF reads some more message/messages from the queue. This essentially locks them and no other agent can read them. They call it pre-reading. This harms the farm scenario very much since while one message is processed other is waiting on the same agent when there are other agents available to process it. That should be fixed in .NET 4.0 RTM. In .NET 4.0 beta 2 its fixed for netMsmqBinding only.

Target Environment

The environment affects the choices you need to make the server farm work. Its configuration choices mostly, but it can bubble up to your service and client code as you will see. Lets consider the target environment:

First there is a “Standard” environment. This includes .NET 3.0, 3.5 with/without SP1, and a mixture of Windows 2003/2008/XP/Vista/Win7 machines.

and then there is this “Modern” environment that includes .NET 3.5/4.0 beta 2 (RTM is not released yet) and Vista/Win7/Server 2008 only machines.

Standard: This means you are using MSMQ 3.0 which means no transactional remote read from the queue. Having at least one agent running on a pre-Vista OS breaks everything and you need to rely on specific architecture to support transactions. But there is a solution. You will need to build Transactional Read-Response. That is something i never did so please share your thoughts of how it shapes-up in real world.

Modern with .NET 3.5 requires you to read with transactions from the remote queue to have any kind or durability. These transactions don’t offer the best performance since you need a transaction even when you are not sure you will be able to handle the message. That’s should be fixed in .Net 4.0 with the new ReceiveContext support and the new Alternative Queuing Model

Modern with .NET 4.0 will become standard after some time. Now it is considered modern. It has the all the goodness - remote transacted read, the ReceiveContext support, No message pre-read, custom dead letter queues etc. It requires again the msmqIntegrationBinding since the directly address formatting issue is still present. Lets hope this will be fixed in the RTM.

In the next post i will present a solution of the scenario we are trying to build. I will show how to fix the pre-read issue on msmqIntegrationBinding that may work on  both .NET 3.5 and .NET 4.0.

If you think layered application design is good and it's the best way to throw some work at the juniors in you team think again.

If you want to learn more about

• Inner Dependency Problem
• Dependency Inversion
• Service locator
• Poor's man Dependency Injection
• Inversion of Control Containers
• Static Gateway Pattern
• What's wrong with singletons
• Boo programming language
• Decorator pattern
• Coding to contract rather than concrete implementation
• Dependency Resolver

there is a very good MSDN article about software component dependencies. I suggest you read it if you haven't already.

Tame Your Software Dependencies for More Flexible Apps (By James Kovacs)

Unfortunately the author missed to mention about Object Builder nor the recently announced Unity Application Block but I think you can investigate further on your own.

Loosen up!
Lubo.

Microsoft made a ton of announcements at MIX08. I just watched Scott Guthrie at the opening key note. He talked about so many things that I didn't remember them all after I finished watching the movie, so I decided to try summarize them here:

• Just Released .NET 3.5, VS2008, IIS7, Windows Server 2008. - I don't know what that means since I am using these for a couple of months now. Well except Longhorn 2008. I think they don't count the MSDN subscription downloads.
• ASP.NET MVC, ASPNET AJAX updated, ASPNET Dynamic Data framework. - Well not yet. Scheduled for "Late this year". I expect this to be this summer since they are talking about couple of releases in that timeframe. Preview releases are available right now though.
• IE8 beta 1.
• CSS 2.1 support at the final IE8 release.
• CSS Certification. They have submitted 702 test cases to W3C test group available at MSDN under BSD license.
• Performance. That's one of the reasons why I switched to Firefox. It is interesting why we don't see these kind of comparisons from Microsoft when they release a new product but we see them just when they are releasing the next version of this exact same product. To tell you the through IE 8 beta 1 is more slower than IE7. But hey its a beta 1 remember?

• HTML 5 Support.
• Better history(Back button) support for AJAX sites through HTML 5 hash functionality.
• Network aware sites through connection events.
• DOM storage. Save pages locally temporary.
• IE developer tools, - Updated. Yeah Firefox has couple of them right now.
• WebSlices - Subscribe to a web page part in IE8. For this to work web sites should specify couple of CSS styles to a div.WebSlices specification is available right now through Open Specification Promise. Samples are also available.
• Silverlight 2 beta  1 available right now. - I installed that immediately. Like the 1,5+ million others do per day.
• Rich High Def Video.
• Adaptive Streaming. Silverlight monitors network bandwidth and automatically adjusts the quality of the video. Very good. I don't know if all those flash video players support that, but in any case this is a must have functionality. Moreover you can plug-in your own algorithm for stream adjustment into the media system.
• Windows Media Services 2008 for Windows Server 2008 - 3x faster, free and already proved to work in real world scenarios.
• IIS7 media pack. - Shipped already. Throttling media buffering download to be only seconds ahead of the current video position.
• Advertising. Making money from your site.
• Silverlight ad template. A wizard for creating advertising for you to include in web pages.
• Atlas AdManager. Manage and track your advertising campaign. Many tracking information. Most importantly how much of the video the user have watched. They call it Engagements. Very detailed really.
• Expression Media Encoder 2. Visually add marker points for dynamically load ads over the video. Very good non intrusion way to show advertisements as overlays on the video the user is watching. XAML overlays also supported.
• NBC Olympics web site is going to be great. - The main issue was that they have less than 150 days to finish it.
• Multilingual support JavaScript, VB, C#, IronPython, IronRubby.
• WPF UI framework support. Layout management. Databinding, Skins and Styles and Animations.
• Networking stacks. REST, SOAP, WS-*, HTTP endpoints, Cross Domain Networking, Raw Sockets support.
• Linq query support.
• Local cache support. That is interesting. Maybe SQL Compact for Silverlight?
• Higher performance.
• Small download 4,3MB.
• Microsoft is partnering with Novel to deliver Linux support for Silverlight. That is great. Most probably it will kill Moonlight project.
• Built in Controls. Available as open source. You can modify them as you see fit. Available already.
• Testing framework for UI controls and for non UI code. 2000 unit tests. Available already as open source.
• Visual Studio 2008 Tools for Silverlight Beta 1 and Expression Blend 2.5 March Preview support. These Expression tools will never come out of beta. Don't you think?
• Silverlight Deep Zoom functionality available for you to use it on your web sites. See it in action on Hard Rock Memorabilia.
• SharePoint Silverlight extensions. Yeah! I am very excited about this specifically, cause man that software will really survive the sun apocalypse. In fact I think this Silverlight thing was invented for this in the first place.
• Mobile devices support. That's huge. How in the world we are going to catch up with iPhone without this.
• Silverlight support on Nokia Symbian S60, S40 and Internet Tablets OS. That's fantastic news! Do you remember the Silverlight's code name? WPF Everywhere anyone? Now what Nokia needs is a nice Singularity OS for mobile devices to get rid of the Symbian+Java crap running on in.
• WPF improvements.
• More controls.
• Streamlined .NET setup. .NET 3.5 is installed by default on Windows Server 2008 but is not installed by Vista SP1 setup.
• Startup performance.
• Graphics Improvements. Offload more onto the GPU. - Will be available as a summer update.
• Custom graphics effects. Onto the GPU also. Custom effects for you to implement. Over pictures, movies and controls.

That is a very long list of announcements. There are at least 15 things I want to test right now but I know I will skip it. As always the time that we have is very limited. The next set of Betas and CTPs are coming very soon. That's how's live on the verge of technology.

Like every windows developer I installed Vista SP1 as soon as it came out on MSDN Subscriptions. I was happy and confident to know that the most of the glitches that bugged me till today will be gone. Service Pack 1 installed without any issues. (I even installed it on my work PC over remote desktop, so the next morning I come to work I wont have to wait for it. Pretty cool).

It was just when I tried to open my blog that I realized it is not working. I got Error 503 Service Unavailable response from the server. A quick look showed that the Application Pool for my blog was failing to serve the first request and logs the following error in Event log.

The worker process failed to initialize correctly and therefore could not be started. The data is the error.

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
 <System>
   <Provider Name="Microsoft-Windows-IIS-W3SVC-WP" Guid="{670080D9-742A-4187-8D16-41143D1290BD}" EventSourceName="W3SVC-WP"/>
   <EventID Qualifiers="49152">2276</EventID>
   <Version>0</Version>
   <Level>2</Level>
   <Task>0</Task>
   <Opcode>0</Opcode>
   <Keywords>0x80000000000000</Keywords>
   <TimeCreated SystemTime="2008-02-21T21:43:34.000Z"/>
   <EventRecordID>36822</EventRecordID>
   <Correlation/>
   <Execution ProcessID="0" ThreadID="0"/>
   <Channel>Application</Channel>
   <Computer>CONROE</Computer>
   <Security/>
 </System>
 <EventData>
   <Binary>05000780</Binary>
 </EventData>
</Event>

So the data is the error. This error in words is 80070005. I was one Windows SDK start away from finding out what that means. I searched for this number in the SDK and found out that this is a Standard COM error "E_ACCESSDENIED 80070005 General access denied error." So I realize I am dealing with a security issue. What was the exact resource that my App Pool was trying to connect and failed? Like Mark Russinovich would do I started Process Monitor to look for more detailed information since the report was totally non comprehensive. Working with Process Monitor is very easy just watch TechEd IT Forum 2007: The Case of the Unexplained... movie where Mark shows some more stuff. The most important rule for Process Monitor is know what you are looking for. In my case that was an Access Denied operation result in the Result column. This would be on a process with credentials that my app pool was configured to work with. So I highlighted all processes that have this user credentials. After that it was easy to identify the operation that failed - the selected row on the screenshot.

The process's that runs my blog had no read access to C:\Windows\System32\inetsrv\config\schema directory. This directory is used by IIS to save its settings. So I changed the security settings to allow my user credentials to read from that directory and voila my blog is up again.

By the way if you watch the The Case of the Unexplained movie you will hear Mark Russinovich say that he strives to remove all hexadecimal error message from user space. I guess Mark and Microsoft as a whole must treat Event log messages the same as user level MessageBoxes. The event log message was not detailed at all and whoever wrote that COM logging should add the event error description not just writing a simple error code to the log.

Recently I have seen couple of articles on the web about how user unfriendly is Windows Vista. This is what provoked me to write about it here. I don't agree with that and that's why I will not provide links to them cause I don't want to increase their Page Rank :)

Now most of these articles are written by journalists that are computer educated people that I don't think are the most common user type that this OS is used by. This doesn't make their opinion absolute. As said they have the knowledge and they probably are the most OS-behavior-sensitive user group that require some attention. So I just wanted to share my thoughts about these passionate negative reviews.

What makes me impression is how these people are used to Windows XP. They always compare feature by feature the new Vista with the "good" old XP. They sometimes compare it with some other Linux like OSes that by default are better than Windows :). Well I think this is the most bad approach to evaluate a new OS. There are many complaints that many things are different in the look and feel, in the behavior, in the minimum hardware requirement etc. While I agree that many of these little things count this doesn't make Windows Vista unfriendly. In fact when writing about it at least write why the user needs to move as quickly as possible to the new OS. All these different things that are bellow the fancy UI (Security, File System, Stability, Performance) makes the new OS the future and there is no going back. I will not go into more details since this will become another useless post about the old versus the new. What I will say is The Queen is dead... long live the King. Change and adopt. That's life.

So let me share some very "unfriendly" thinks about Windows Vista.

What I found yesterday is a nice information regarding daylight savings. .

Also I am able to add two additional clocks so I am able to track all the Webcasts that I am going to hook up to. And if you don't know it you can click on the month title and "ride the century" to find a specific date

How user unfriendly is that!