1

I am working on a command processing application which uses azure service bus queue. Commands are issued from a website and posted to the queue and the queue messages are processed by a worker role. Processing involves fetching data from db and other sources based on the queue message values and sending it to different topics. The flow is ,

  1. Receive message
  2. process the message
  3. Mark message as complete / Abandon message On processing exception.

The challenge I face here is the processing time. Sometimes it exceeds the maximum message lock time period (5 minutes -configured) and hence the message is unlocked and it re-appears for the worker role to pick up (consider multiple instances of the worker role). So this causes same message to be processed again.

What are the options I have to handle such a scenario.?

I have thought about ,

  1. Receive message - add to a local variable - mark message complete. In case of exception send the message again to the queue or to a separate queue (let us say failed message queue). A second queue also means another worker role to process it.

  2. In the processing there is a foreach loop that runs. So I thought of using a Parallel.Foreach instead . but not sure how much of time gain it will give and also read some posts on issues when using Parallel in azure.

Suggestions,fixes welcome.

Improve this question

3 Answers 3

Reset to default
1

Aravind, you can absolutely use SB queue in this scenario. With the latest SDK you can renew the lock on your message for as long as your are continuing to process it. Details are at: http://msdn.microsoft.com/en-us/library/microsoft.servicebus.messaging.brokeredmessage.renewlock.aspx

This is similar to the Azure storage queue functionality of updating the visibility timeout: http://msdn.microsoft.com/en-us/library/windowsazure/microsoft.windowsazure.storage.queue.cloudqueue.updatemessage.aspx

Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

Looks a good option. Guess one has to have a timer and renew this lock. but currently stuck with azure sdk 1.7.
Irrespective of which Azure SDK version you are using, you can always reference the latest Service Bus client library using the following NuGet package: nuget.org/packages/WindowsAzure.ServiceBus
0

You may want to consider using an Azure Queue, the maxium lease time for an Azure Queue message is 7 days, as opposed to the Azure Service Bus Queue lease time of 5 minutes.

This msdn article describes the differences between the two Azure queue types.

If the standard Azure Queue doesn't contain all the features you need you might consider using both types of Queue.

Improve this answer

2 Comments

I had considered both storage and servicebus queues and then arrived at sb queue.
If your processing is going to take over 5 minutes then the sb queue doesn't look like a great fit for your problem, the storage queue allows for much much longer processing time. Your suggested solution 1 doesnt look great, as if the role/network goes down after you mark it complete then you will never get the chance to re-add it to the queue. If the service bus features are essential to you, consider removing message from sb queue, add it to storage queue, then delete it from sb queue, you can then process message from storage queue (on same role if you want)
0

You can fire off a Task with a heartbeat operation that keeps renewing the lock for you while you're processing it. This is exactly what I do. I described my approach at Creating a Task with a heartbeat

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.