Python - Azure function service bus trigger batch processing

Question

I am using Azure function service bus trigger in Python to receive messages in batch from a service bus queue. Even though this process is not well documented in Python, but I managed to enable the batch processing by following the below Github PR.

https://github.com/Azure/azure-functions-python-library/pull/73

Here is the sample code I am using -

function.json

{
  "scriptFile": "__init__.py",
  "bindings": [
    {
      "name": "msg",
      "type": "serviceBusTrigger",
      "direction": "in",
      "cardinality": "many",
      "queueName": "<some queue name>",
      "dataType": "binary",
      "connection": "SERVICE_BUS_CONNECTION"
    }
  ]
}

__init__.py

import logging

import azure.functions as func
from typing import List

def main(msg: List[func.ServiceBusMessage]):
    message_length = len(msg)
    if message_length > 1:
        logging.warn('Handling multiple requests')

    for m in msg:
       #some call to external web api

host.json

"version": "2.0",
  "extensionBundle": {
    "id": "Microsoft.Azure.Functions.ExtensionBundle",
    "version": "[3.3.0, 4.0.0)"
  },
  "extensions": {
    "serviceBus": {
      "prefetchCount": 100,
      "messageHandlerOptions": {
        "autoComplete": true,
        "maxConcurrentCalls": 32,
        "maxAutoRenewDuration": "00:05:00"
      },
      "batchOptions": {
        "maxMessageCount": 100,
        "operationTimeout": "00:01:00",
        "autoComplete": true
      }
    }
  }
}

After using this code , I can see that service bus trigger is picking up messages in a batch of 100 (or sometimes < 100) based on the maxMessageCount but I have also observed that most of the messages are ending up in the dead letter queue with the MaxDeliveryCountExceeded reason code. I have tried with different values of MaxDeliveryCount from 10-20 but I had the same result. So my question is do we need to adjust/optimize the MaxDeliveryCount in case of batch processing of service bus messages ? How both of them are related ? What kind of change can be done in the configuration to avoid this dead letter issue ?

MaxDeliveryCountExceeded is normally because of a failed message processing. Are you certain you are not throwing any errors anywhere in your actual code, like if there is an error calling the API? — Alex
– Alex, Commented Feb 10, 2023 at 7:24
@AlexAIT yes we are getting 502 error from backend api which is a python web app running in linux app service plan . It is happening if we are sending more than 10 messages in a batch . Is there any limit of maximum no of concurrent http request to a linux app service in Azure? Can we increase that? — Niladri
– Niladri, Commented Feb 10, 2023 at 8:28
@AlexAIT Also we are getting the bellow error in service bus trigger The lock supplied is invalid. Either the lock expired, or the message has already been removed from the queue , I am using python so there is no option to manually complete the message if I set autocomplete:false — Niladri
– Niladri, Commented Feb 10, 2023 at 8:37
The lock error indicates that you are not processing the whole batch within maxAutoRenewDuration. My guess would be that your API has an issue or is fairly slow. If you replace the api call with a one second delay then I expect no more errors. You then need to tweak the settings and the API. — Alex
– Alex, Commented Feb 10, 2023 at 14:34
You can also try increasing the duration. If you have a batch of 100 messages with 4 seconds per message, this will be more than the current maxAutoRenewDuration of 5 minutes. The 1 second delay was just to prove that the API is the problem. Don't call the API, just simulate it to show that batch processing works. — Alex
– Alex, Commented Feb 10, 2023 at 16:38

Alex · Accepted Answer · 2023-02-11 22:10:27Z

From what we discussed in the comments, this is what you encounter:

Your function app is fetching 100 messages from ServiceBus (prefetchCount) and locking them for a maximum of maxAutoRenewDuration
Your function code is processing messages one at a time at a slow rate because of the API you call.
By the time you finish a batch of messages (maxMessageCount), the lock already expired which is why you have exceptions and the message gets redelivered again. This eventually causes MaxDeliveryCountExceeded errors.

What can you do to improve this?

Reduce maxMessageCount and prefetchCount
Increase maxAutoRenewDuration
Increase the performance of your API (how to do that would be a different question)
Your current code would be much better off by using a "normal" single message trigger instead of the batch trigger

PS: Beware that your function app may scale horizontally if you are running in a consumption plan, further increasing the load on your struggling API.

Collectives™ on Stack Overflow

Python - Azure function service bus trigger batch processing

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related