Understanding the Windows Azure AppFabric Service Bus [QuotaExceededException]

This post provides insight into some of the undocumented details of the Windows Azure AppFabric Service Bus connection quotas. In particular, it answers the following questions: Under what circumstances will you receive the QuotaExceededException when establishing Service Bus connections? What are the “real” Service Bus connection quotas? What can you do to overcome them? And, finally, are there any plans to address these quota limitations?

THIS ARTICLE HAS BEEN UPDATED AND THE QUOTA’S MENTIONED BELOW NO LONGER APPLY. FOR MORE INFORMATION ON THE CURRENT QUOTA LIMITATIONS PLEASE REFERENCE THE UPDATED POST.

Overview

There is no magic in interpreting the Service Bus QuotaExceededException message. It is not cryptic, and the basic information is provided in the exception message itself: “User quota was exceeded. Maximum number of connections per solution exceeded.” This is not too difficult to comprehend… you exceeded some maximum value (i.e. the quota) and the Service Bus cannot process your request. The challenge with the exception is in understanding what quota you exceeded, and what options you have for dealing with this. The background for this post is based on a recent customer engagement where we started receiving this message as the deployed solution exceeded the maximum number of allowed Service Bus connections. However, this came as a surprise, because the FAQ on Windows Azure Pricing states the quota is 500 Service Bus connections, and in this particular test, the solution only had one-tenth of that being consumed, or about 50+ connections, before we started seeing this error.

For reference, here’s the actual error that was encountered:

[QuotaExceededException] User quota was exceeded. Maximum number of connections per solution exceeded.; STACK: Server stack trace: at Microsoft.ServiceBus.RelayedSocketInitiator.Connect(Uri uri, TimeSpan timeout) at Microsoft.ServiceBus.Channels.BufferedConnectionInitiator.Connect(Uri uri, TimeSpan timeout) at Microsoft.ServiceBus.Channels.ConnectionPoolHelper.EstablishConnection(TimeSpan timeout) at Microsoft.ServiceBus.Channels.ClientFramingDuplexSessionChannel.OnOpen(TimeSpan timeout) at Microsoft.ServiceBus.Channels.CommunicationObject.Open(TimeSpan timeout) at Microsoft.ServiceBus.Channels.CommunicationObject.Open(TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout) at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout) at …

Understanding the Current Limits

After a few calls to support, and some collaboration with the engineers, it appears there are some undocumented quotas based on the size of your connection pack. Figure 1 represents the connection packs currently offered (as of Apr. 2011) and the “actual” per namespace quota limitations imposed on each package size. This clearly illustrates the distinction between the connection pack size (i.e. your discounted pricing based on number of connections) and the associated quotas (i.e. the maximum number of concurrent connections allowed before you start getting the QuotaExceededException).

Figure 1 – Quotas by Connection Pack Size

Given this newfound information, it was easy to determine why we started encountering the aforementioned exception. In our case, we were using the “pay as you go” model, and reached the internal limit of 50 connections.

How to Overcome the Current Quota Limits

Now that we have a basic understanding of why we were encountering this exception, as well as an understanding of what the quotas are, we can evaluate the options at our disposal for overcoming these limits. Here are two options:

1. Upgrade your connection pack size until you get to a level where you will not be exceeding the quotas. This can be accomplished by navigating to the new Service Bus management portal, which has recently been integrated into the overall Windows Azure Management Portal experience.

2. What if you know you will have more than 835 connections (i.e. the current limit for the 500 connection pack)? In this case, you can work with the Azure Business Desk and request a “custom” connection pack size (i.e. > 500) and a “custom” quota (i.e. whatever you need this to be!). That’s right, in addition to requesting a new connection pack size, you can (and should) request a new quota/connection limit. You can request this by calling the phone number appropriate for your region, which can be found by completing the questions at our Microsoft Support for Windows Azure site.

Note:

Based on the current pricing plan (April 2011), if you exceed the number of concurrent connections (as defined by your connection pack size), you will be charged at the “pay as you go” rate for each additional connection. It is only after you reach the quota/limit that you will receive the QuotaExceededException message.

Plans to Address the Quota Limits

For those of you that have plans to exceed the 835 current connection limit, or for those of you that want to continue a “pay as go” model without upgrading your connection pack size for the sole purpose of increasing your quota, you will be happy to know that a fix has been planned and the product group will introduce a new quota limit. The new quota limit (which should be available within the next few weeks/months) is expected to be 2000 concurrent connections, regardless of the connection pack used! While the process for overcoming the quota limit will likely remain the same, you should be happy to know that you should experience fewer QuotaExceededException messages as a result of this planned change.