python and kafka KeyError: -1

Question

Python kafka does not work. Any reason why? I mean I cant event connect to kafka and i get an error? Is there a kafka client that works with 0.8? I mean this is a brand new server. I just booted it up.
I am using https://github.com/mumrah/kafka-python

from kafka.client import KafkaClient
kafka = KafkaClient(kafka_domain, 9092)


Traceback (most recent call last):
  File "/home/ubuntu/workspace/rtbhui-devops/simulations/pixel_druid_simulations.py", line 36, in <module>
    kafka = KafkaClient(kafka_domain, 9092)
  File "/usr/local/lib/python2.7/dist-packages/kafka/client.py", line 38, in __init__
    self.load_metadata_for_topics()  # bootstrap with all metadata
  File "/usr/local/lib/python2.7/dist-packages/kafka/client.py", line 247, in load_metadata_for_topics
    self.topics_to_brokers[topic_part] = brokers[meta.leader]
KeyError: -1

In kafka logs I see the below.

[2014-02-26 08:36:21,471] INFO Closing socket connection to /222.127.xxx.xxx. (kafka.network.Processor)
[2014-02-26 08:40:30,801] ERROR [KafkaApi-1393401480] Error while fetching metadata for partition [topic-pixel,0] (kafka.server.KafkaApis)
kafka.common.LeaderNotAvailableException: Leader not available for partition [topic-pixel,0]
    at kafka.server.KafkaApis$$anonfun$17$$anonfun$20.apply(KafkaApis.scala:474)
    at kafka.server.KafkaApis$$anonfun$17$$anonfun$20.apply(KafkaApis.scala:462)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
    at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61)
    at scala.collection.immutable.List.foreach(List.scala:45)
    at scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
    at scala.collection.immutable.List.map(List.scala:45)
    at kafka.server.KafkaApis$$anonfun$17.apply(KafkaApis.scala:462)
    at kafka.server.KafkaApis$$anonfun$17.apply(KafkaApis.scala:458)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:206)
    at scala.collection.immutable.Set$Set1.foreach(Set.scala:81)
    at scala.collection.TraversableLike$class.map(TraversableLike.scala:206)
    at scala.collection.immutable.Set$Set1.map(Set.scala:68)
    at kafka.server.KafkaApis.handleTopicMetadataRequest(KafkaApis.scala:458)
    at kafka.server.KafkaApis.handle(KafkaApis.scala:68)
    at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:42)
    at java.lang.Thread.run(Thread.java:744)

Me too! Did you perchance delete a topic? I saw this when I deleted all topics with the kafka commandline tool. — RickyA
– RickyA, Commented Mar 27, 2014 at 9:09
I think this happens when leader election is not performed correctly. I had the error when kafka-topics.sh --describe said the leader was 1, it disappeared when it became 0 again. — RickyA
– RickyA, Commented Mar 27, 2014 at 9:27

dpkp · Accepted Answer · 2015-05-25 17:13:23Z

1

This is fixed in kafka-python version 0.9.0. The error is internal to Kafka Python, not the kafka server, and simply involves handling partitions that are currently without a leader (which happens for any new topic when you auto-create topics, but other than that is fairly rare under normal operations)

See

https://github.com/mumrah/kafka-python/pull/109

edited May 25, 2015 at 17:13

answered Mar 28, 2014 at 5:04

dpkp

1,4699 silver badges15 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

python and kafka KeyError: -1

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related