I need to process incoming messages on Azure. Each message will be associated with a specific entity – say, through an EntityId property – and messages belonging to the same entity must be processed in-order with respect to each other. At the same time, I would to preserve the serverless aspect of Azure Functions; if I have steady streams of messages for 1,000 entities, I'd like to have 1,000 concurrent executions of my function. I haven't found a clean way of achieving this. Service Bus queues have sessions, which are the closest implementation of my requirements, but they are not supported in Azure Functions: https://github.com/Azure/azure-functions-host/issues/563. However, they seem to be supported in Azure Logic Apps. I'm thinking of creating an Azure Logic App that is triggered by a Service Bus queue using sessions ("Correlated in-order delivery using service bus sessions" template), then hooks to an HTTP-triggered Azure Function for processing messages. The Logic App's only purpose is to prevent multiple messages belonging to the same entity/session from being processed concurrently. Can someone provide some insight on whether this approach would work, and whether it has any caveats?
-
You can try to "drive" your workflow by your entity (entityid) and create according messages (jobs) in a queue, one for each entity. The job than collects all needed messages (from wherever they are stored) (best way to have them in a partition by entityid) and than process them in order. You can than drive your workflows in parallel. Durable Functions would be a candidate for that.Sebastian Achatz– Sebastian Achatz2018-12-02 12:58:59 +00:00Commented Dec 2, 2018 at 12:58
Add a comment
|
1 Answer
Have a look at this article from one of the Azure Function Team Member: In order event processing with Azure Functions
It uses Azure Functions and Azure Events Hubs:
Azure Events Hubs can handle billions of events, and has guarantees around consistency and ordering per partition.
For your scenario, each message related to the same EntityId has to go to the same partition.
The trick to have in order processing and let azure function scale independently is by pulling batches from event hubs, and preserve order.
You function should looks like this:
[FunctionName("EventHubTrigger")]
public static async Task RunAsync([EventHubTrigger("ordered", Connection = "EventHub")] EventData[] eventDataSet, TraceWriter log)
{
log.Info($"Triggered batch of size {eventDataSet.Length}");
foreach (var eventData in eventDataSet)
{
try
{
// Process message in order here.
}
catch
{
// handle event exception
}
}
}
I would recommend you to read the whole article, it is very instructive
You can find the whole solution on Github:
https://github.com/jeffhollan/functions-csharp-eventhub-ordered-processing
10 Comments
Þórir Sachie
Thanks for sharing this! I'd looked into that article, but I don't think it'd address my scenario, since Event Hubs only support up to 32 partitions (Partitions). This limits parallelism to 32, whereas I'd like to be able to scale to 100s or 1000s of entities.
Thomas
No worries :-) but why would you want to scale to an instance per entity ? Do you have any other concerns ? Not sure but using logic app you will also reach limits because you can only execute 300 000 actions per 5 mins and it could become very expensive
Þórir Sachie
My concern is performance. Each entity will have a steady stream of messages which I'd like to process as soon as possible. I wouldn't want an entity's messages to be queued behind other messages belonging to other entities that happen to be on the same partition. Thanks for letting me know about the Logic App limits; that's the information I'm trying to gather right now.
Þórir Sachie
You were right; not only is there the 300,000-action limit, but there's also a trigger concurrency limit of 50, which makes Logic Apps almost as bad as Event Hubs for lack of scalability.
Thomas
You should not be worried too much about performance. In the article they processed 100 000 messages in few seconds. Also do you need to send infomation back to the user ?
|