Questions tagged [message-queue]
A message queue is a service that receives events or messages from producers and provides them to consumers.
259 questions
3
votes
3
answers
557
views
Design question for exactly-once processing in a message-driven system using a unique ID
To achieve exactly-once processing where messages are consumed from a queue with at-least-once delivery, many sources (e.g. here and here and here) suggest attaching a unique ID to messages in the ...
- 139
0
votes
5
answers
170
views
How can I measure which messages are causing queue backups?
Consider the following multi-producer, multi-consumer system:
For simplicity, assume that we only have one queue
Messages are dequeued on a first-in-first-out basis
Oldest-message-pickup-latency has ...
- 119
3
votes
3
answers
259
views
Handling rate limits / delays in consumers without affecting performance of other operations
I have a producer that generates a batch job that consists of multiple operations (approx. 100 - 10000). These operations can be processed in any order, ideally as fast as possible.
The processing of ...
- 141
7
votes
5
answers
450
views
What can I do to get a message processor to slow down the rate of writes that it is making to a database?
We have this architecture:
queue -> message processor (horizontal scaling) -> RDBMS
Sometimes external systems dump 10k messages onto the queue and the message processor of course dutifully ...
- 229
2
votes
4
answers
311
views
Ensure concurrent processing based on a message key
I've been wondering about the following: Say you have an async message/task and you want that message only processed one at the time. for example: Only process one order for each customer at the time. ...
- 174
0
votes
1
answer
94
views
Design a sequential processing of records
About 10 to 15 records are processed per day with the time interval of 5 minutes between each record.
System A inserts a record in DB and sends id of that record to active mq.
System B Listener ...
- 119
2
votes
0
answers
139
views
parent correlation id in message with rabbit queue and open telemetry
I have a rabbit queue that I use to communicate between applications.
I have a chain of three message.
App 1 generate and push a message in rabbit(message 1)
App 2 catch the message and generate ...
- 307
0
votes
1
answer
152
views
Is RabbitMQ suitable for allowing SaaS customers to subscribe to events?
I work on a SaaS system which is generally a single-tenant environment. Customers (whom the SaaS instances are for) would like a way to subscribe to events, such as updates of certain data entities. ...
- 101
0
votes
0
answers
113
views
Event based update of In memory Hashmap in multiple pods of kubernetes
I have a service (S1) that is deployed on kubernetes having multiple replicas (pods).
S1 requires some data from database (Cosmos Cassandra Api) for calculation.
This data is constantly updated by ...
- 101
1
vote
1
answer
104
views
Handling a single event as broadcast and round-robin depending on service
I have Service A that publishes an event to RabbitMQ. I have two instances of Service B that will use the event to write to its database. I have two instances of Service C that will send the event ...
- 136
0
votes
1
answer
654
views
Is a message queue the right pattern to use for intra-module communication in a .net core application
Background Information
We have a .Net core application that uses Hangfire (pretty version of cron) jobs to batch process different types of data.
As a bogus example, we have one job that will run ...
- 581
0
votes
2
answers
284
views
Complicated task scheduling architecture
So I want to make an application in which a user will hit an endpoint to save a job model to storage that includes some metadata to perform a long computation against which will be offloaded to a ...
- 101
2
votes
2
answers
9k
views
Is Kafka an event bus or a publisher subscriber (pubsub)pattern?
Building on this answer here, and its comments it entails that subscribers need to know and locate the publishers in a traditional pub/sub system. It also entails that publishers need to live at least ...
- 131
0
votes
2
answers
338
views
How can I track the status of multiple messages in asynchronous processes?
Let's say I've a sendMessage endpoint. This endpoint looks at configured message destinations and then puts 1..N messages into a queue (e.g. {userId: 3, type: SMS, trackingID: X} and {userId: 3, type: ...
user374484
0
votes
1
answer
166
views
How to seamlessly interact with a message broker when the underlying system (SQS, RabbitMQ) can change depending on the environment?
Having a poor knowledge of the extended features of the available message brokers, I was wondering how to approach this.
We have some environments where only RabbitMQ is available, others where ...
- 111
0
votes
1
answer
63
views
Which techniques to follow to make a fast and consistent message consumer that writes to a RDBMS [closed]
I am building a consumer application that reads messages from a queue and runs basic sanitization and deduplication and persists them to an RDBMS.
The functional requirements are
messages must not be ...
- 131
2
votes
2
answers
3k
views
Implement message queue in API Gateway
Recently we have this client who asks for a API Gateway solution, in this case it’s Kong. Currently, they have 10 services (200 APIs) that are running on really legacy stuff (built with C++ and ...
- 129
0
votes
1
answer
74
views
Queue system with priority on Consumers
Imagine a queue system with different consumers - some are low cost whist (like 'reserved/spot' instances in the AWS world) others would be high cost.
The system should ensure the messages are ...
- 320
0
votes
0
answers
67
views
How to handle maintenance to your queuing server (e.g. rabbitMQ)
I'm new to queuing and looking into options that could be done operationally. I'm sorry if I have any conceptual misunderstandings related to queuing.
I watched this video where the presenter made a ...
- 341
1
vote
0
answers
483
views
Do we really need a message brocker in WebSocket system?
I'm engineering architecture of a new web based software.
I've never worked on high-scale softwares before and I'm reading a lot about it.
To increase client-side speed and reduce load on servers, and ...
- 19
-1
votes
1
answer
790
views
Message queuing systems: sequential processing with dependencies between queues
I am trying to understand Rabbitmq and how I can accomplish sequential message processing in certain circumstances only. For example if I have 2 queues. One for pricing an order and one for sending ...
- 107
0
votes
1
answer
136
views
Expose api to third parties to send PDF files (invoices)
Our system needs to process PDF invoice files from generic external third parties.
Using HTTP is the most obvious, but this isn't the greatest of ideas in case we need to deal with bulk uploads.
Even ...
- 185
0
votes
2
answers
228
views
Prevent missing updates using a message broker after creating a task
I'm trying to design a data updates mechanism in my micro-services architecture.
For the sake of simplicity, let's assume we have two micro-services A and B, B exposes an API for creating some tasks, ...
- 109
0
votes
0
answers
48
views
Design a Program which Forwards Work Pieces to Corresponding Modules
I am working on a program that will listen to a queue of incoming message and will process each message differently depending on the metadata of that message. The messages must be processed in order ...
- 101
4
votes
1
answer
6k
views
Event bus vs PubSub
I have been using queues and PubSub patterns for years but never really came across the terminology "event bus". After some googling I didn't really find a clear distinction between "...
- 189