Where the Upsert Kafka connector consumer start?

Question

I'm trying to read from kafka in upsert mode. Everything works fine. But I noticed that scan.startup.mode is not supported.

Is my assumption right that in upsert mode kafka connector reads the whole topic from earliest offset by design or there is another logic? If I need to reduce amount of processed data should I reduce retention of topic?

Flink version: 1.14.6

David Anderson · Accepted Answer · 2024-08-22 11:06:26Z

1

Reading from the beginning is necessary for correctness. You can compact the topic to reduce the effort and storage required.

answered Aug 22, 2024 at 11:06

David Anderson

44.3k4 gold badges41 silver badges73 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

jinzuchi · Accepted Answer · 2024-08-22 11:13:09Z

0

Yes, scan.startup.mode is not supported.It specify as earliest-offset when create org.apache.flink.streaming.connectors.kafka.table.KafkaDynamicSource instance.

// always use earliest to keep data integrity
StartupMode earliest = StartupMode.EARLIEST;

edited Aug 22, 2024 at 11:13

answered Aug 22, 2024 at 9:20

jinzuchi

1481 silver badge6 bronze badges

2 Comments

Niko Over a year ago

I'm on Flink 1.14 and I'm speaking about UPSERT mode. You sending me the links to APPEND mode.

jinzuchi Over a year ago

sorry for not notice it, I had update my answer

Collectives™ on Stack Overflow

Where the Upsert Kafka connector consumer start?

2 Answers 2

Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related