We have built real time data processing pipelines on kafka topics in the past with kafka streams technology. But we were always limited by the number of partitions on the kafka topic for concurrency because one partitions cannot be assigned to more than one thread. In this particular use case the ordering of messages across different partitions didn't matter. We will always end up having free CPU resources that would virtually do noting while getting bottlenecked on network I/O and creating a backlog on the topic. So, we ended up moving our service to reactive framework Quarkus.
But with the concept of virtual threads introduced in Java 21, it seems like we shall be able to overcome the bottleneck on network I/O because a lot more virtual threads can now do the network I/O with kafka concurrently and hence allowing our service to get records faster and process them faster. As mentioned in the Java 21 virtual thread documentation we are trying to increase throughput and not latency.
Has anyone tried using kafka streams with Java virtual threads and if yes, how was the performance?
