15 questions
0
votes
1
answer
251
views
Unable to install Karafka gem on M1 Mac
I'm trying to add Karafka to an existing Ruby project and running into Architecture issues while trying to install the gems on my M1 Mac. I've faced this issue with other gems in the past but my ...
- 1,524
1
vote
2
answers
192
views
Karafka on AWS Fargate with MSK: Process Hangs When Consuming Messages One at a Time
I'm running a Karafka application that consumes messages one at a time inside an ECS container on AWS Fargate, using Amazon MSK as my Kafka broker.
Setup Details:
Karafka configuration and code:
class ...
- 80
1
vote
1
answer
744
views
Scaling karafka consumer instances for high traffic kafka topics
I'm using Rails with Karafka gem. In one of the topic the traffic will be huge, so i added 10 partitions to the topic, also i set the Karafka::App.config.concurrency to 10. But this isn't giving me a ...
- 65
1
vote
1
answer
670
views
Why Karafka Server Throw error query watermark offset?
I'm using this gem for my Ruby on Rails Apps to produce and consume message from kafka server.
https://karafka.io/docs/
and this is my karafka.rb file
# frozen_string_literal: true
class KarafkaApp &...
- 39
1
vote
1
answer
454
views
Is Karafka compatible with applications other than Rails? i.e Sinatra
I have a service that publishes a Kafka event whenever a user attribute is updated, now to consume and process this event I have a gem that uses Karafa. Every application that boots the gem will be ...
- 13
0
votes
0
answers
489
views
kafka is designed to have many producers
We have been testing kafka (cloud-based) to implement a data transmission system of many producers (approximately 27000 linux machines), a consumer (spring kafka listener) and a topic with 10 ...
0
votes
1
answer
972
views
Long delays between processing of two consecutive kafka batches (using ruby/karafka consumer)
I am using karafka to read from a topic, and call an external service. Each call to external service takes roughly 300ms. And with 3 consumers (3 pods in the k8s) running in the consumer group, I ...
- 3,391
0
votes
1
answer
2k
views
Karafka start_from_beginning not working as expected
I've created a project to help with my understanding of Kafka. It's set up as three identical Rails apps all inside Docker with Karafka configured to consume the messages - if you create a record in ...
- 987