0

We are using Spring Boot Kafka listener to consume messages from Kafka. Currently, we have the auto commit offset enabled and so the offset commits happen every 5 seconds (default auto commit interval). We are polling Kafka for a batch of 100 records at a time and take about 300 milliseconds to process the messages. In a regular scenario where there will be multiple polls of messages and one commit every 5 seconds as they get processed fairly quickly. Even if the service pod goes down before the next offset commit, the code is idempotent so any messages not committed yet will be republished/reprocessed in the event of this issue.

However, I am thinking of a way to handle a remote scenario of the batch processing taking more than the default window of 5 seconds in which case the offset would be committed before the processing is completed. If the processing is completed fine, then there is no issue. but if there is any error with processing these messages which are already marked consumed, we will lose those messages.

I am looking for ways to have this handled without losing the messages. I was thinking of changing the ack mode to manual and commit after every batch no matter how long it takes (of course less than the max.poll.interval) but this will increase the # of offset commits by 10 fold. I also thought of having an internal timer to acknowledge when the timer elapses say 5 seconds since the last poll. This way the commit won't happen when the current poll processing takes longer than 5 seconds and will be done only later.

However, I am wondering if this leaves a possibility of we receiving a bunch of messages and they will be waiting to be committed until the next poll and if this is the last set of messages, then they will be waiting for a long time. I have also tried different ack modes like COUNT/COUNT_TIME but none of that are solving the use case here. Not sure how this can be handled.

Any thoughts on this?

Improve this question

2 Answers 2

Reset to default
1

If you're using Spring Kafka containers (e.g. @KafkaListener as a basis for your stuff in Spring Boot) - you don't need manual acknowledgement.

Just set AckMode in your Container Properties to RECORD - and be happy.
Container would do that lower-level Kafka API Consumer manual ack for you.

P.S. On a side note - the default 5 sec is way, way, WAY too long in a nowadays nanosecond-fine world. For one, the default for "native" Kafka API is 100ms, to my recollection.
I don't even know what these Spring Kafka guys were thinking when they set it (although it goes along with a messy quality of the package itself).

Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

RECORD will commit on every record no? What i am trying to do is poll for batch of messages (100 or 200) and then commit them at regular intervals. I was thinking of just committing them at the end of every batch manually but that will be too many commits to Kafka compared to now once every 5 seconds. Already we are seeing some issues with Kafka stability where 6 pods are polling messages in batches of 100 which i feel is not a lot so wondering if there would be any performance impact of doing this for every batch immediatedly.
Yeah, every single one - and what's wrong with it? "Too many commits" is not a thing, I absolutely can't get what's your concern with it? You have any evidence it impacts anything? Especially, it impacts anything more than potential re-consumption of the whole batch 100 or 200 batch given your processing time per message is quite substantial (whole 300ms)?
1

You are on the right track here but with a slight misunderstanding. Not sure how you think it would increase it by 10 fold..

I was thinking of changing the ack mode to manual and commit after every batch no matter how long it takes (of course less than the max.poll.interval) but this will increase the # of offset commits by 10 fold

You don't have to commit all the messages in the batch. you only need to commit the last offset of the batch. Which means 1 commit per batch. If the way you process messages makes it lose the order, then you could retrieve the "max" offset of a batch and commit that at the end of batch processing. This approach is what I have used and works well.

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.