Azure durable function for bulk insert toward Postgres DB

Question

my Azure's serverless architecture is composed from some producers, a service bus, un azure function and a Postgres DB (see the below picture).

The Postgres DB is a legacy requirement, I cannot change it.

The following is the operations flow:

The producers send a message with a temporal frequency towards service bus (about 9000 messagges/minute). Every producers sends a single message.
The azure functions consumes the message and insert a row into the Postgres DB

In order to avoid a strong load of DB and open a lot of connections I would to aggregate the messages into the function and inset them by bulk insert. Could I work well with a durable function (entity function)?

Can you help me please? Best Regards

Each function invocation should receive a batch of messages anyway already and not being called per single message github.com/Azure/azure-functions-servicebus-extension/issues/… — silent
– silent, Commented Feb 27, 2020 at 10:55
Looking at the PR, I would say: yes. but not sure how exactly. Might be worth to open a issue in the github repo. default seems to be 1000: github.com/Azure/azure-functions-servicebus-extension/pull/39/… — silent
– silent, Commented Feb 27, 2020 at 13:03

silent · Accepted Answer · 2020-02-27 13:38:04Z

1

Ok, got it to work as discussed in the comments. You don't need a durable function but you should receive (and then write) messages in batches from the Service Bus. Here is a batching example:

[FunctionName("QueueTriggeredFunction")]
public static void Run([ServiceBusTrigger("demofunctionqueue", Connection = "queueconstring")]string[] myQueueItems, ILogger log)
{
    log.LogInformation("Received messages {count}", myQueueItems.Length);
    foreach (var myQueueItem in myQueueItems)
    {
        log.LogInformation($"C# ServiceBus queue trigger function processed message: {myQueueItem}");
    }
}

host.json to set the maximum number of messages per batch

{
  "version": "2.0",
  "extensions": {
    "serviceBus": {
      "batchOptions": {
        "maxMessageCount": 200
      },
      "messageHandlerOptions": {
        "maxConcurrentCalls": 1
      }
    }
  }
}

edited Feb 27, 2020 at 13:38

answered Feb 27, 2020 at 13:33

silent

16.5k7 gold badges61 silver badges114 bronze badges

Sign up to request clarification or add additional context in comments.

12 Comments

Hugo Barona Over a year ago

He is using consumption plan. So, even if you limit the messages per batch on the function's instance, you will not limit the number of instances that can scale up on his function app. So he would have 10 instances of the function app consuming 1-2 messages each.

silent Over a year ago

Fair enough, I edited the host.json above to add the config switch for max concurrency

Hugo Barona Over a year ago

maxConcurrentCalls does not work with Consumption plan. Please read my comments on my answer.

silent Over a year ago

According to this one it should work? github.com/Azure/azure-functions-host/issues/…

silent Over a year ago

yes, should be like that. If you still see multiple instances being spawned, you might need to also set WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT as suggested by @HugoBarona

|

Hugo Barona · Accepted Answer · 2020-02-27 09:56:53Z

1

I would consider Durable Functions just in case if you have multiple steps, so then you can build your workflow using Durable Functions. In your case, what I would do is to play around with the maxConcurrentCalls parameter on your hosts.json file, so you can then set an acceptable value for concurrent calls for your DB. Please reference this sample to get more info.

answered Feb 27, 2020 at 9:56

Hugo Barona

1,39611 silver badges23 bronze badges

2 Comments

Michel Foucault Over a year ago

Hi Hugo, thanks for you reponse. Does "maxConcurrentCalls" limits the number of concurrent calls towards consumers (Azure function)?

Hugo Barona Over a year ago

@MichelFoucault in that case you should use the configuration setting WEBSITE_MAX_DYNAMIC_APPLICATION_SCALE_OUT. Read more here

Collectives™ on Stack Overflow

Azure durable function for bulk insert toward Postgres DB

2 Answers 2

12 Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

12 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related