Ruby Kafka Producer BufferUnderflow

Question

I wanted to use the ruby kafka client library for producing events, but have come up against a problem that I am not sure how to solve. Any help would be appreciated.

I have tried using kafka-rb (acrosa, mheffner and bpot forks). The problem is that no matter what I send to it via the library e.g.

require 'kafka'
host = 'localhost'
port = 9092
producer = Kafka::Producer.new(

        :topic => 'login',
        :host => host,
        :port => port
)
producer.send([Kafka::Message.new("aaaaa")])

I get a:

java.nio.BufferUnderflowException
    at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
    at java.nio.ByteBuffer.get(ByteBuffer.java:675)
    at kafka.api.ApiUtils$.readShortString(ApiUtils.scala:22)
    at kafka.api.ProducerRequest$.readFrom(ProducerRequest.scala:34)
    at kafka.api.RequestKeys$$anonfun$1.apply(RequestKeys.scala:34)
    at kafka.api.RequestKeys$$anonfun$1.apply(RequestKeys.scala:34)
    at kafka.network.RequestChannel$Request.<init>(RequestChannel.scala:48)
    at kafka.network.Processor.read(SocketServer.scala:321)
    at kafka.network.Processor.run(SocketServer.scala:231)
    at java.lang.Thread.run(Thread.java:680)

on the server. On the same server I can send text through the provided console producer without any issues.

If you have seen this before I would appreciate the help. As I am not very familiar with Scala, I am not sure what the problem is, but it seems to me that the line where this exception is thrown, has to do with reading the clientId from the socket, and it also seems to me that there is no such thing sent from the ruby client.

When I look at the messages produced on tcpdump form kafka-rb and the provided producer. The ruby ones seem shorter. Additionally it does not matter whether I use kafka-0.7 or 0.8, i get the exact same behaviour.

It turns out that I was using 0.8 in all my attempts as the kafka-rb does not support the 0.7 format. Thanks for the help. — oarsome
– oarsome, Commented Feb 11, 2013 at 16:05

prenomenon · Accepted Answer · 2013-02-08 15:19:18Z

1

I could reproduce your error with Kafka 0.8 but when I try this implementation:

require 'rubygems'
require 'kafka'
producer = Kafka::Producer.new({ :host => "localhost", :port => 9092, :topic => "test" , :compression => 0 })
message = Kafka::Message.new("some random message content")
producer.send(message)

it works with Kafka 0.7:

consumer example

According to the Kafka documentation, the client you mentioned is only supported for 0.7.x (https://cwiki.apache.org/KAFKA/clients.html#Clients-Ruby). The 0.8 section says:

The 0.8 release changes the protocol fairly substantially. This version hasn't been released yet. The new protocol is documented here. A number of clients are in progress, and we will update when they are complete.

So I think https://github.com/acrosa/kafka-rb will not work in this case :-(.

Best

pre

answered Feb 8, 2013 at 15:19

prenomenon

1305 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

oarsome Over a year ago

Looks like I was trying to use kafka-rb with 0.8 even when I thought I was using it with 0.7 (which works). Thanks

cevaris Over a year ago

Mind you the built in Zookeeper does not work in 0.7

Collectives™ on Stack Overflow

Ruby Kafka Producer BufferUnderflow

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related