Spring-Kafka RetryableTopic causes RecordTooLargeException

Question

I am using spring-kafka RetryableTopic for non-blocking retries with a fixed BackOff and a single retry topic (https://docs.spring.io/spring-kafka/reference/html/#single-topic-fixed-delay-retries)

I noticed that I get RecordTooLargeException when retry-attempt is relatively high, and while checking the message I see that it contains the Kafka headers for all previous attempts, and some headers like kafka_exception-stacktrace they are quite heavy.

Why does it try to publish the retry message with the headers of the previous retries? I can not find any configuration for that. Can these headers somehow be manipulated to cut them before publishing?

Gary Russell · Accepted Answer · 2021-11-03 20:33:15Z

2

Good point; I opened an issue.

https://github.com/spring-projects/spring-kafka/issues/1994

I'll see if I can come up with a work-around until this is implemented.

EDIT

public class HeaderStrippingInterceptor<K, V> implements ProducerInterceptor<K, V> {

    @Override
    public void configure(Map<String, ?> configs) {
    }

    @Override
    public ProducerRecord<K, V> onSend(ProducerRecord<K, V> record) {
        Header header = record.headers().lastHeader(KafkaHeaders.EXCEPTION_STACKTRACE);
        if (header != null) {
            record.headers().remove(KafkaHeaders.EXCEPTION_STACKTRACE);
            record.headers().add(header);
        }
        return record;
    }

    @Override
    public void onAcknowledgement(RecordMetadata metadata, Exception exception) {
    }

    @Override
    public void close() {
    }

}

Then add its class name to the producer configs; for example, with Spring Boot:

spring:
  kafka:
    producer:
      properties:
        "[interceptor.classes]": com.example.demo.HeaderStrippingInterceptor

edited Nov 3, 2021 at 20:33

answered Nov 3, 2021 at 20:04

Gary Russell

176k14 gold badges168 silver badges194 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Gary Russell Over a year ago

I added a work around; see the edit.

Collectives™ on Stack Overflow

Spring-Kafka RetryableTopic causes RecordTooLargeException

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related