628 questions
1
vote
1
answer
188
views
IIDR CDC Kafka Custom KCOP disable schema auto register
we are trying to replicate Db2 changes from z/OS to Confluent Kafka.
Everything works out fine for the production of simple JSON records in Kafka.
As our policies for Kafka Topics and Schemas are ...
- 76
0
votes
0
answers
37
views
about EMQX Enterprise 6 Action Sink to Kafka cluster
I am a newbie in EMQX architecture. Recently, for test purposes, I deployed an EMQX enterprise 6 single node, with Kafka producer connector integration (Action Sink).
The compatition matrix of the ...
0
votes
1
answer
131
views
Why am I getting a null row when querying a CDC function in an ADF Lookup activity?
I am trying to build an ADF pipeline that incrementally updates a table using Change Data Capture (CDC). And I'm running into an issue when I try to use a CDC built in table function in an ADF Lookup ...
- 1
2
votes
1
answer
127
views
Debezium + Flink Oracle CDC - "db history topic or its content is fully or partially missing" for some tables
I am using Flink with Debezium to consume CDC changes from Oracle DB tables via LogMiner.
For some tables, everything works fine. For example, the following table works without issues:
CREATE TABLE ...
- 23
0
votes
2
answers
82
views
how have CDC on redis?
I'm using CDC services like (Debezium) on my Mongo or Postgres but somehow I came up with situation that I need have to CDC on Redis . for example get streams of event occur in Redis like adding new ...
0
votes
0
answers
53
views
MongoDB custom CDC handler using PySpark
I want to replicate a collection and sync in real time.
The CDC events are streamed to Kafka and I’ll be listening to it and based on operationType (insert/delete/update) I’ll have to process the ...
- 197
2
votes
0
answers
21
views
In windows, I can't see usb cdc descriptions
I'm developing a new project by using stm32 MCU and USB is one of most important features in my project. But unfortunately I always see manufacturer as "Microsoft" and comport name as "...
0
votes
0
answers
97
views
Error 300 VIEW SERVER STATE permission when trying to disable CDC as the SA
Having an issue with the CDC process. I'm trying to add a column to a CDC table and the first step is to disable the current CDC table. I can't get past this step even though I'm logged in as 'sa'. ...
- 83
2
votes
1
answer
653
views
Read incremental data from iceberg tables using Spark SQL
I am trying to read incremental data between two snapshots
I have last processed snapshot (my day0 load) and below is my code snippet to read incremental data
incremental_df = spark.read.format("...
- 33
0
votes
1
answer
42
views
Triggers execution order inside transaction for change tracking
I have a Firebird table Orders with bunch of columns. I have after update triggers that log table information in events_history table in the following manner:
SET TERM ^ ;
CREATE OR ALTER TRIGGER ...
0
votes
0
answers
37
views
2 Stage Synchronizer confusion
In this picture, Ds goes metastable during the second rising edge of CLK-B because Din changed values during its aperture time at the first rising edge of CLK-B. However, Ds can go metastable during ...
0
votes
1
answer
48
views
Event Ordering Across Tables in Kafka
In Kafka, it's generally recommended to map one table to one topic. However, what if the order of events across multiple tables is important for a specific key, such as CompanyId?
For example, ...
- 108
0
votes
0
answers
53
views
How to Implement Maxwell CDC with AWS MSK?
I am trying to implement Change Data Capture (CDC) using Maxwell's Daemon with AWS Managed Streaming for Apache Kafka (MSK). My goal is to capture changes from a RDS MySQL database and stream them to ...
0
votes
1
answer
121
views
Unable to delete a CDC resource in ADF
I had created a CDC resource in ADF to test out, how it works. After few tests, It became non-functional with a red cross mark against it. I cannot edit or view the activity anymore.
If I try to ...
1
vote
2
answers
236
views
AWS DMS: Same S3 bucket as target for multiple SQL sources
I cannot find information about how to achieve what's in the title of this question.
Let's say I have different SQL databases from different departments in the org and I want to migrate all of them to ...
- 597
1
vote
0
answers
199
views
Kafka Debezium Oracle Source Connector Intermittent Lag & Data Delay
I am working with Kafka, and we are using Debezium to capture CDC (Change Data Capture) data from an Oracle source database. Below are my Debezium configuration parameters:
{
"name": &...
- 155
0
votes
1
answer
207
views
USB CDC freezes (Host to Device transfers)
My setup
I try to use USB FS Host on GD32F405 (=STM32) microcontroller. I use CMSIS and operate directly on registers.
I have Android device, that is working as VCP (Virtual COM port) by default and ...
- 202
0
votes
1
answer
156
views
Amazon S3 as Source for Apache Flink using CDC
I need to publish the event using CDC whenever a file is added to S3, to Apache Flink, from Flink the newly added file has to be read and processed.
- 505
1
vote
1
answer
345
views
SQL Server CDC: cdc.fn_cdc_get_all_changes_ returns a row containing all null values - LSN = 0x00000000000000000000
I'm running the following query on a SQL Server capture instance:
DECLARE @from_lsn binary (10), @to_lsn binary (10)
SELECT @from_lsn = sys.fn_cdc_get_min_lsn ( 'dbo_ABC' );
SELECT @from_lsn;
...
- 5,472
-1
votes
1
answer
602
views
Issues with executing PySpark / Delta MERGE statement due to special character in a table name
I am working on implementing CDC (Change Data Capture) which will use Azure Sql as the source and Azure Databricks delta lake as the target.
Azew Sql creates CDC tables with a very specific column ...
- 15.7k
0
votes
0
answers
166
views
DMS Source Data and Latency spike nightly
I'm using a SQL Server on-prem source for a DMS to replicate to a SQL Server RDS. Every night, at about midnight, I'm seeing a spike of ~1.6M CDC incoming changes on my DMS task, but nothing is making ...
0
votes
1
answer
709
views
CDC capture job not running after running sys.scan_job manually
I have a CDC enabled table used for replication. One of my stored procedure I have, runs this exec sys.sp_cdc_scan manually for a specific case (my stored procedure is invoked by a DDL trigger which I ...
- 312
0
votes
1
answer
160
views
How to stop clean up scheduler in Azure SQL DB CDC
I have requirement where i need to perform Audit of data for that i have planned to use Azure DB change data capture(CDC).
In Azure CDC it works on scheduler unlike on premise where SQL Server Agent ...
2
votes
0
answers
331
views
Debezium automatic topic creation not working as expected
I'm working with CDC using a debezium connector managed by Confluent.
I was trying to use the topics creation group feature offered by debezium, to create topics automatically based on the size of ...
- 794
0
votes
2
answers
446
views
I do not find the cdc.fn_cdc_get_net_changes_*
I am developing SQL Server business logic using CDC.
The CDC works fine, I see the changes in cdc tables but I cannot find function cdc.fn_cdc_get_net_changes*. But other functions exist: cdc....
- 6,592