Questions tagged [apache-kafka]
Apache Kafka is publish-subscribe messaging rethought as a distributed commit log.
99 questions
2
votes
4
answers
730
views
How to design a distributed system with an event broker where strong ordering is required?
I was trying to model a fairly simple real world model inside a distributed system and got stuck thinking about timing and order and would appreciate some external view on it.
Assuming I have this ...
- 769
0
votes
1
answer
336
views
Best Practices for Designing Kafka Message Processing with Independent Tasks
I am designing a message processing system for Kafka topic A.
Each message has a message_type and needs to query DB to determine the tasks required for this message_type. The tasks are independent of ...
- 117
-2
votes
1
answer
234
views
Transaction Management Across Microservices with Rollback Mechanism using Kafka
I am currently developing a microservices architecture consisting of two services, and I'm facing a challenge related to transaction management and error handling.
Microservice A: This service is ...
- 1
3
votes
2
answers
452
views
In Hexagonal Architecture how do I implement a Dead-Letter Queue?
I have an application using Hexagonal Architecture. I want to implement a DLQ in 2 scenarios:
When the event streaming adapter's incoming message is unreadable (invalid payload, etc)
When the domain ...
- 133
1
vote
1
answer
106
views
How to test integration with an infinite generator
I'm struggling to (integration) test a component that makes use of an infinite generator.
Background:
At a really high level, this component is essentially a "middleman" in a kafka workflow. ...
- 271
0
votes
2
answers
242
views
Seeking a Third Opinion on Kafka Consumer Implementation and Architectural Disagreements
I've been at my current job for about 4-5 months, mainly working with Go, and I have no prior experience with Kafka. Before this, my background was in JavaScript, Node.js, React, etc. I recently got a ...
- 274
2
votes
2
answers
195
views
Using json objects (with maintained schema in registry) vs. string vs. hash as message key
Folks,
I am wondering if there is any dis/advantage to using json objects as message keys.
Assume I have 2 event streams coupled with 2 tables in a source system. Each of those tables have its own ...
- 31
1
vote
1
answer
508
views
KRaft design architecture - Separation of Controllers and Brokers
We want to install Kafka Kraft cluster on 35 machines ( machines are DELL physical machines - type R760 And OS is Linux RHEL 8.6 version )
Since we are dealing with Kafka production cluster , we want ...
- 111
0
votes
0
answers
94
views
Using messaging and rest to communicate services at the same time
I have two microservices, USERS and PRODUCTS, the second one creates objects referred to first one so it needs first to check that exists
First I though in a redis cache, but then I realize that for ...
2
votes
2
answers
9k
views
Is Kafka an event bus or a publisher subscriber (pubsub)pattern?
Building on this answer here, and its comments it entails that subscribers need to know and locate the publishers in a traditional pub/sub system. It also entails that publishers need to live at least ...
- 131
0
votes
1
answer
341
views
What pattern should I use for aggregating different types of events in Apache Kafka?
I'm trying to use Kafka for my event handling, but I'm struggling a bit to understand what pattern I should use.
My use case is this:
One consumer waits for one type of event.
When that event is ...
0
votes
2
answers
755
views
Message broker design pattern best practice [closed]
I'm designing the architecture of a platform to introduce a message broker in an existing data collection web application.
This web application is currently used to upload data from excel files that ...
- 109
1
vote
0
answers
137
views
Micro-Services: How to avoid repeated polling of the same event from Kafka
I have an Micro-services where we get the event messages from a Kafka messaging system. The event message contains the orders information, quantity, the mode of package and delivery location. At a ...
- 111
0
votes
1
answer
569
views
How to integrate a Python server with Kafka?
I have the request to integrate my Python server application (Flask/waitress) with Kafka.
That means that it should frequently poll a certain Kafka topic (on the order of minutes) and process all new ...
- 119
2
votes
1
answer
166
views
Data Ingest Architecture Advice
I have a requirement where we need to collect N different events and store them for analysis. I am having trouble coming up with a general architecture for this.
FINAL REQUIREMENTS
The end goal of the ...
- 29
2
votes
1
answer
1k
views
Handling Errors in a Microservice architecture and asynchronous message queue
When inter-communicating through http requests between Microservices it is quite easy to reply with error messages to users, you can just catch the error of a different Service and forward it to the ...
- 139
0
votes
2
answers
492
views
Event driven architecture conditional logic
As this article explains the way to deal with persistent state in an event log driven architecture is to have some "microservice" subscribed to events and be responsible for maintaining that ...
0
votes
2
answers
338
views
How to publish single event from multiple services
Let's say I have a system that parses network traffic, and generates event of things that happened.
example:
ip1 was talking to ip2
ip2 was talking to ip3
etc
Now let's say that these events are ...
- 101
0
votes
1
answer
684
views
Solution architecture for a Kafka streaming to website system
Not sure if this is the correct forum for this question - but I could really use some advice.
I need to design a system (within certain constraints).
The system must implement the following logic:
A ...
2
votes
2
answers
2k
views
What is the advantage of Kafka connector to MongoDB over using MongoDB change streams?
I need to stream changes from my MongoDB instance to Elasticsearch. I also need to perform complex transformations on changed documents. After doing some research I narrowed my options down to 2:
...
- 167
0
votes
1
answer
667
views
Mixing an API server and a background worker
What are your thoughts on having a service/process be both an API server and a background worker, consuming messages off a queue like SQS?
I have typically kept these two separate to keep single ...
- 111
8
votes
2
answers
16k
views
Is Kafka needed in a realtime chat application?
I'm developing a realtime chat application with an Angular frontend and Java backend. I've found a couple of examples that resemble what I am trying to achieve, such as:
https://medium.com/swlh/...
- 221
1
vote
1
answer
410
views
How would a Search API microservice fit in an event driven architecture?
I apologize if this sounds really strange. I am very new to System Design and have just been reading up on Event driven systems. To gain better understanding and to perfect my thought process I ...
- 157
1
vote
1
answer
322
views
Options for building initial event sourcing projection when you cannot go back to the beginning of the history
I’m building a microservice which depends on a set of downstream services to collect the data necessary to operate. It is a data delivery system, I’m not source of truth for any data, my system rather ...
0
votes
0
answers
49
views
Designing sane messaging paradigm for concurrent updates
We have a weird use-case that I need to support in my application and needed some opinions on how to design it.
At a basic level the application allows users to work on a "list" of "tweets" she cares ...
- 109