0

I'm trying to consume messages from kafka topic using Azure durable function and as the new messages arrive I want to invoke the activity function to process the messages. The problem here is that the callback function isn't able to access the context and other local variables which are needed to invoke the activity function.

Please suggest if I'm dong it correctly and if there are any better/alternate approaches.

Here is my function code:

/* the orchestrator is invoked by a timer trigger */
/* using kafka javascript SDK (@confluentinc/kafka-javascript) to create a consumer */

const kafkaConsumerOrchestratorName = 'kafka_consumer_orchestrator';
const kafkaConsumerActivityName = 'kafka_consumer_activity';

df.app.orchestration(kafkaConsumerOrchestratorName, function* (context) {
    yield kafkaService.consumer.run({
        eachMessage: async ({ topic, partition, message }) => {
            /* context not accessible here */
            yield context.df.callActivity(kafkaConsumerActivityName, { topic, partition, message });
        }
    });
});

df.app.activity(kafkaConsumerActivityName, {
    handler: async (input, context) => {
        /* process messages */
        console.log(input);
    }
});
2
  • Your Kafka consumer in the orchestrator, leads to scope issues with context. A better approach is to separate Kafka polling into a timer-triggered function and use the orchestrator to manage the workflow. But according to what I read, Azure Functions Kafka trigger is likely a better choice if you have access to this. Commented Apr 1 at 6:57
  • @mplungjan thanks for the suggestions, I started off with the azure function kafka trigger but one of my use cases is to be able to seek kafka messages which is not available in azure function kafka trigger. I already have an open question for the same here: stackoverflow.com/questions/79543236/… Commented Apr 1 at 7:09

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.