120 questions
2
votes
0
answers
42
views
MongoDb Kafka Source Connector Pipeline is causing BsonInvalidOperationException
I have a collection which I need to covert to timeseries using mongodb kafka connector. The source collection have document of this shape
{
"_id": "6798680c885bb2a8fbe93485",
...
- 639
0
votes
0
answers
38
views
Use value of _id as key for MongoDB Kafka Source Connector
My collection looks like this
Collection
And the source configuration looks like this
curl -X POST "http://localhost:8083/connectors" \
-H "Content-Type: application/json" \
...
1
vote
0
answers
173
views
Kafka Connector Timestamp converter is not working when use with SMT
I've to used timestamp converter with SMT "io.debezium.connector.mongodb.transforms.ExtractNewDocumentState"
but message in topic unix timestamp fields not convert to format that i want
here ...
0
votes
1
answer
224
views
How to setup and configure mongodb kafka soruce connector using aws msk connect?
I have setup a MSK cluster using public subnets. I also have turned on public access for this MSK cluster. The cluster is up and running. I am trying to setup MongoDB Kafka Source Connector using MSK ...
- 167
0
votes
1
answer
192
views
I am trying to find a way to prevent truncation of resume token
I am trying to find a way to prevent truncation of resume tokenwhen error is thrown by mongodb server for 16 mb size limit. I am reading resume token to make sure documents over 16 mb is read properly ...
1
vote
1
answer
179
views
Dynamic WriteStrategy for MongoDB Sink connector
Is there a way to dynamically choose the WriteStrategy of a MongoDB sink connector?
Use case at hand -
Read from a topic and write to mongo atlas using mongodb sink connector in a way that the ...
- 401
1
vote
1
answer
672
views
Confluent Kafka Timestamp Converter value issue
I am using Apache Kafka with Confluent Connect v7.3.2 platform and a MongoDB Connector deployed as Sink Connector in order to stream messages with 2 timestamp fields into a Collection. These values ...
- 395
0
votes
0
answers
367
views
Unknown field on documentdb streams (changeStream.fullDocumentBeforeChange)
I am trying to integrate document db (5.0.0) with kafka redshift connector. when doing so i keep on getting this issue where it complains that '$changeStream.fullDocumentBeforeChange' is an unknown ...
- 419
0
votes
1
answer
497
views
Mongodb-kafka connector failing while connecting MSK to Mongodb
Im trying to connect MongoDB with AWS Managed kafka using the source connector (downloaded from Confluent Hub), but while creating AWS connector im getting the error message:
Code: InvalidInput....
- 1
0
votes
0
answers
372
views
MongoSourceConnector tombstone didnt work
I want make a CQRS with data mesh, and i already success to create MongoSourceConnector and ElasticsearchSinkConnector, but im confuse why my MongoSourceConnector configuration didn't publish null ...
- 1
0
votes
0
answers
80
views
I have some data in MongoDB that I need to migrate to another MongoDB, but _id is getting stored in different format
Legacy MongoDb is having _id in below format
legacy mongodb id
New MongoDB is having _id in below format
_id: "{"$oid": "57f2773113b28328a0dabfe4"}"
How to make data ...
0
votes
0
answers
653
views
Kafka mongoDB connector sink don't consume any events
I have a mongoDB Kafka connector that doesn't consume any event anymore.
My configuration is the following:
{
"name": "event-mongodb-sink",
"config": {
"...
- 2,000
0
votes
1
answer
230
views
MongoSinkConnector failed to start "the configured document ID strategy all records are required to have keys which must be either maps or structs"
Mongo Sink Connector failed to start with below error:
With the configured document ID strategy, all records are required to have keys, which must be either maps or structs.
Record Key String Format
...
0
votes
1
answer
104
views
Mongosink Connector to exclude topice
I have mongo sink connector for my kafka cluster which sinks topic data into mongo database.
I am looking for properties to exclude one topic in my sink connector if there is any.
Explanation:
local....
- 19
0
votes
1
answer
118
views
Creating a MongoDB collection for each Kafka topic automatically in MongoDB Kafka Sink Connector
Is there a way in the MongoDB Kafka Sink Connector to create a MongoDB collection for per Kafka topic by defining a pattern or prefix or other way?
- 1,584
1
vote
1
answer
473
views
Kafka connect can't find my custom write strategy
I am trying to implement a custom write strategy for a sink connector that writes to mongodb as per the documentation here:
https://www.mongodb.com/docs/kafka-connector/current/sink-connector/...
0
votes
2
answers
698
views
How to group kafka topics in different dbs and collections with mongodb sink connector depending on kafka topic name or message key/value
As the title states, I'm using debezium Postgres source connector and I would like MongoDB sink connector to group kafka topics in different collection and databases (also different dbs to isolate ...
- 1
0
votes
1
answer
612
views
MongoDB Kafka Source Connector - copy.exisintg.pipeline config
I'm running a MongoDB Kafka source connector (official mongodb connector version 1.7.0), and defining both pipeline and copy.existing.pipeline properties in order to filter some columns- see below ...
- 764
0
votes
2
answers
1k
views
Kafka connect- Mongodb Sink Connector
I am new to kafka connector. I have been explore about it about a week. I have used create and update the mongodb via mongodb connector curl commands. I am bit struggling to understand the concept and ...
- 137
0
votes
1
answer
137
views
How to connect already setup kafka cluster to mongodb?
How can I connect kafka events to a mongodb sink?
The resources I found on the net using confluent they make a cluster for you and didn't find how to connect my already existing cluster
- 15
0
votes
1
answer
793
views
MongoDB Kafka Connect - Sink connector failing on updates
I am new to Kafka connect.
I am trying sync the change stream from 1 mongo collection to another using Kafka connectors, both Inserts and updates operations
Source config-
{
"name": "...
- 11
1
vote
0
answers
402
views
MongDB sink connector: How to use a field from json as _id
I'm trying to ingest json data from Kafka topic to MongoDB using MongoDB-Kafka-Connect and have the following properties configured
topics=example
connector.class=com.mongodb.kafka.connect....
- 161
0
votes
1
answer
2k
views
Use Kafka connect with AWS documentDB
I'm trying to use AWS DocumentDB as a sink for storing data received from Kafka and was wondering if the MongoDB Kafka connector works with DocumentDB as its documentation mentions that it is ...
- 161
0
votes
2
answers
994
views
Mongo Kafka Connector Collection Listen Limitations
We have several collections in Mongo based on n tenants and want the kafka connector to only watch for specific collections.
Below is my mongosource.properties file where I have added the pipeline ...
0
votes
1
answer
958
views
multiple collections mongodb to Kafka topic
The application writes data every month to a new collection (for example, journal_2205, journal_2206). Is it possible to configure the connector so that it reads the oplog from the new collection and ...
- 141