1,144 questions
1
vote
1
answer
62
views
Understanding Debezium Snapshot
Debezium’s PostgreSQL snapshot docs note :
always
The connector performs a snapshot every time that it starts. The snapshot includes the structure and data of the captured tables. Specify this value ...
- 42.1k
0
votes
1
answer
75
views
Setup Kafka Consumers to process Debezium snapshot before current events
I am working with Java/Spring/Kafka on a new system that will consume events from both compacted topics and standard non compacted CDC topics. The events on these topics are published by a Debezium ...
- 1,991
0
votes
1
answer
120
views
Custom Debezium image falling with FileNotFoundException: /opt/kafka/custom-config/log4j.properties
I am trying to get debezium up and running on EKS. However i keep running into errors.
Below is the dockerfile which i have created to build the image and push it to ECR
FROM quay.io/strimzi/kafka:...
0
votes
0
answers
36
views
Debezium initial snapshot on demand
I have a debezium 3.0 in my python project. Postgres-debezium connector has the following structure:
{
"name": "dbz_name",
"config": {
"connector.class&...
0
votes
0
answers
47
views
Restart Debezium by clicking a button on the web application In the Kubernetes environment
I have 3 pods running in a Kubernetes environment. The three pods are as follows:
(1) Web application pod
(2) Oracle database
(3) Debezium
The web application is connected to an Oracle database, ...
- 5,212
2
votes
1
answer
127
views
Debezium + Flink Oracle CDC - "db history topic or its content is fully or partially missing" for some tables
I am using Flink with Debezium to consume CDC changes from Oracle DB tables via LogMiner.
For some tables, everything works fine. For example, the following table works without issues:
CREATE TABLE ...
- 23
0
votes
0
answers
52
views
Debezuim ui create connector
I have the following docker-compose yaml:
version: '3'
networks:
kafka-network:
driver: bridge
services:
zookeeper:
image: confluentinc/cp-zookeeper:7.6.0
environment:
...
0
votes
0
answers
63
views
Unable to ingest MongoDB data with its schema in s3 using Kafka debezium connectors
I am trying to ingest data from MongoDB to S3 in parquet format, along with the schema registered in the schema registry. The Setup is as follows:
MSK -> EC2(EKS)
Below is my MongoDB Source ...
- 641
0
votes
0
answers
98
views
how to solve debezium Kafka Connect vulnerability
Our team noticed that the current Debezium images (e.g., debezium/connect) still include Apache Kafka 3.9.0, which is affected by several recently disclosed security vulnerabilities:
CVE-2025-27817
...
- 33
0
votes
1
answer
137
views
KafkaConnector fails to auto-create topics while KafkaConnect creates its own successfully (Strimzi + Debezium + Avro + Oracle)
I'm working in a Kubernetes environment where we deploy Kafka and related services using a custom Helm chart. Here's the stack:
Kafka via Strimzi Operator: 0.46.0, deployed in KRaft mode, version 4.0....
0
votes
2
answers
111
views
MongodDB Debezium connector issue with partial avro schema
I am encountering issue when trying to CDC data from MongoDB source to kafka using Debezium connector with confluent schema registry. I only want the CDC data to have some of the fields which are ...
0
votes
1
answer
195
views
Kafka Connect 4.0.0 and Debezium 2.7
I get the following error when I run kafka when i run connect-standalone.sh. I've ensured that I'm using Kafka 4.0.0, as well as having the correct jars for debezium in the plugins directory. I've ...
- 354
0
votes
0
answers
66
views
Dead letter storage and failure recovery in Debezium standalone server with HTTP sink
I have set up Debezium's standalone server and configured it to use HTTP client as sink and SQL Server as source, the problem is that if the HTTP server is not responding, the entire Debezium server ...
0
votes
1
answer
157
views
Debezium convert types
I have a MySQL column which should store a boolean value. My Django driver models this with a TINYINT value. I want to replicate this data in a PostgreSQL but there this column is a BOOLEAN.
The ...
- 178
1
vote
0
answers
156
views
Debezium ad hoc snapshot not working because of no maximum key
I'm having problem trying ad hoc snapshot in debezium postgres source connector. I already created the signalling table and add the signal table to my debezium config.
When I tried to insert new row ...
- 21
0
votes
0
answers
109
views
How to Set Table-Specific Ordering Key in Debezium for Google Pub/Sub Sink?
I'm using Debezium Server to capture changes from a PostgreSQL database and publish them to Google Cloud Pub/Sub. I want to set a table-specific ordering key so that messages related to the same ...
- 193
0
votes
0
answers
175
views
Debezium RabbitMQ outbox event static routing key
I am trying to implement outbox pattern by reading inserts in a table from a MySQL and transforming them into events in RabbitMQ.
My Outbox table contains:
id
uuid
payload
exchange
routing_key
...
0
votes
2
answers
329
views
Postgres CDC failure and WAL filling up disk space?
Suppose a CDC data loading tool (Fivetran, Airbyte, Debezium, etc) gets into a failure state and is not successfully running its incremental syncs of a Postgres data source. Also suppose that ...
0
votes
0
answers
107
views
Debezium mysql connector is capturing schema changes for tables that are not in the table include list
I am using Debebezium CDC with the outbox transformer and we need it to start up as fast as possible. Since we are using the outbox, there is only ever going to be one table that we are interested in (...
- 1,478
1
vote
1
answer
87
views
Custom transform for Kafka Connect
I get the below error:
[2025-02-10 14:54:53,530] INFO AbstractConfig values:
(org.apache.kafka.common.config.AbstractConfig:370)
[2025-02-10 14:54:53,639] ERROR Failed to create connector for .\kafka-...
- 756
0
votes
0
answers
68
views
Debezium problem with Array bidimensional
I have a table with an _integer field. It is a two-dimensional array, and when Debezium reads all the fields, this field returns null. For example, the data:
{{23966},{8301}}
How can I solved this ...
- 191
0
votes
0
answers
150
views
Kafka Connect Sink Connector
I am using kafka connect with debezium jdbc sink.
My config is
"config": {
"connector.class": "io.debezium.connector.jdbc.JdbcSinkConnector",
"table.name.format&...
- 157
0
votes
0
answers
57
views
How to Reduce Metadata Queries in Debezium JDBC Sink Connector?
I’m using the Debezium JDBC Sink Connector to ingest data from a Kafka topic into PostgreSQL. The database is experiencing high CPU utilization, primarily due to frequent metadata queries made by the ...
- 3,633
0
votes
0
answers
729
views
DebeziumException: Error processing row in XXXX, internal schema size 39, but row size 2 , restart connector with schema recovery mode
I have an set-up with Kafka connector using debezium(2.7) and After updating into debezium as 3.0.
A insertion of new row into table working as expected but Facing below errors, while tries to update ...
- 610
0
votes
0
answers
159
views
Getting Error: trying to setup connection with GCP managed kafka service with Debezium using docker via compose.yaml?
[org.apache.kafka.clients.admin.AdminClientConfig]
debezium-connect-managed-new | 2024-12-20 18:01:24,017 INFO || Retrieved token with principal sub [org.apache.kafka.common.security.oauthbearer....
