Azure ServiceBus - same message read multiple times

Question

We have some issues with messages from Azure ServiceBus being read multiple times. Previously we had the same issue, which turned out to be due to lock timeout. Then, as the lock timed out the messages were read again, and their deliveryCount increased by 1 for each time the message was read. After this, we set the max delivery count to 1 to avoid resending of messages, and also increased the lock timeout to 5 minutes.

The current issue is a lot more strange.

First, messages are read at 10:45:34. Message locks are set to 10:50:34, and deliveryCount is 1. The reading says it succeeds, at 10:45:35.0. All good so far.

But then, at 10:45:35.8, the same messages are read again! And the delivery count is still 1. Both the sequence number and message id are the same in the two receive logs. This happens for a very small percentage of messages, something like 0,02% of the messages.

From what I understand, reading a message should either result in a success where the message should be removed, or an increase of deliveryCount, which in my case should send the message to DLQ. In these cases, neither happens.

I'm using ServiceBusTrigger, like this:

    [FunctionName(nameof(ReceiveMessages))]
    public async Task Run([ServiceBusTrigger(queueName: "%QueueName%", Connection = "ServiceBusConnectionString")]
        string[] messages,

This seems to be like a bug in either the service bus or the library, any thoughts on what it could be?

@ThiagoCustodio I don't think sessions is enabled. Are sessions prerequisite for "transactional" reads? — Tobb
– Tobb, Commented Nov 3, 2021 at 22:13
@ThiagoCustodio did you find a resolution / root cause of the issue? I observe the same issue - deliveryCount is always 1 but the same message triggers an Azure Function multiple times. — Inako
– Inako, Commented Feb 6, 2024 at 1:47
@Inako is it possible that you're not completing the message? — Thiago Custodio
– Thiago Custodio, Commented Feb 6, 2024 at 15:01

Sean Feldman · Accepted Answer · 2021-11-03 14:41:22Z

0

That’s not the SDK but rather the specific entity. It sounds like the entity is corrupted. Delete and recreate it. If that doesn’t help, then open a support case.

On a different note, most of the time when delivery count is set to 1 is an indicator of something off. If you truly need at-most-once delivery guarantee, use ReceiveAndDelete mode instead of PeekLock.

answered Nov 3, 2021 at 14:41

Sean Feldman

26.3k9 gold badges63 silver badges86 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Tobb Over a year ago

I have to receive each message exactly once, as each message represents money. And there are lots of them. The issue here is that they are read twice (and a very few times 3) and then deleted, but then we have duplicates in the database. I would implement some mechanism to make message consumption idempotent, but as a temporary fix we set delivery count to 1, since having duplicates is worse than having messages on DLQ. In other MQs I have used, duplication of messages would be a serious bug. Don't know enough about Azure Service Bus to say anything about the standards there though.

Sean Feldman Over a year ago

ASB doesn't duplicate messages. Duplicates might be received by the competing consumers as a result of failing to process within 5 minutes. Either way, I'd have to understand and see your architecture to suggest anything more specific. If not the entity, I have a strong feeling it's the design of the process that's causing the problem. The devil is in the details 🙂

Tobb Over a year ago

I'm pretty sure that our setup isn't optimal :p But still, I can't seem to understand how message consumption could lead to anything other than the message being removed or the delivery count being increased, just seems like something that shouldn't be possible.

Collectives™ on Stack Overflow

Azure ServiceBus - same message read multiple times

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related