I have a producer which writes messages to a topic/partition. To maintain ordering, i would like to go with single partition and I want 12 consumers to read all the messages from this single partition(no consumer group, all the messages should go to all consumers). Is this achievable? I read some forums that only one consumer can read per partition.
Add a comment
|
2 Answers
You may use SimpleConsumer to achieve exactly what you are asking - no consumer groups, all consumers can read a single partition. However this approach means you have to handle offset storing and broker failure handling yourself.
Another option is to use high level consumer with different consumer groups (you could just assign a random UUID to each consumer). This way you'll be able to consume one topic/partition with all consumers and be able to commit offsets and handle broker outage.
The rule "only a single consumer can consume a topic/partition" applies only to consumer groups, e.g. only one consumer IN GROUP can consume one topic/partition simultaneously.
4 Comments
skumar
if i use unique consumer groupid for each consumers, I will read the all messages for each customer from single partition? ie something like in code.. props.put("group.id", KafkaProperties.groupId);
serejja
Yes, if your topic with a single partition has 1000 messages in it then all consumers with different consumer groups will read 1000 messages each.
skumar
Can you please give some examples for the approach1 (handling offset and broker failer) and for approach 2 (assign randow uui to each consumer)?.
serejja
Regarding the first case it's too broad to write - that highly depends on whether you are supposed to commit offsets at all and be durable on broker failures (e.g. what should happen when broker dies). In general I'd recommend to use the second approach and just use a groupid like this:
props.put("group.id", UUID.randomUUID().toString())If you have multiple consumers on the same partition, it beats your initial requirement of maintaining ordering. Though, you would have ordered storage, but the consumption would be un-ordered. Do make sure you are really OK with that. If yes, you can just treat each consumer as a different consumer group.
