Unable to get the list of topics from AWS MSK using Python

Question

we are trying to get the list of topics using the Python, but it returns empty list.

from kafka.admin import KafkaAdminClient

#configure Kafka client for SCRAM

client = KafkaAdminClient(
bootstrap_servers="b-2-public.url.url2.c3.kafka.eu-west-3.amazonaws.com:9196",
sasl_mechanism="SCRAM-SHA-512",
sasl_plain_username="user",
sasl_plain_password="pass",
security_protocol="SASL_SSL")

topics = client.list_topics()

print(topics)

we are able to retrieve the list of topis using the Java but for some reason not in Python. we are using python 3.7, kafka-python 2.0.2.

Any ideas?

@brushtakopo in a connection string or in .aws/credentials file? I would expect any kind of connection timeout error if the code will not be able to establish a connection — rholdberh
– rholdberh, Commented Mar 27, 2023 at 7:58

Krupesh Patel · Accepted Answer · 2025-01-30 06:49:13Z

0

First, double-check your config. Sometimes, even a small mistake in SCRAM authentication can cause issues. Since Java is able to fetch the topic list but Python isn’t, it’s likely either a configuration issue or the Python Kafka client isn’t connecting properly.

If you're running this inside an EC2 instance within the same VPC, you might not even need authentication. Try removing security_protocol="SASL_SSL" and see if it works without it.

Also, check your SSL config—maybe there's a problem with TLS or missing certificates? Try running it with security_protocol="SASL_PLAINTEXT" or PLAINTEXT just to see if the connection works.

Reference: AWS MSK Authentication Guide

answered Jan 30 at 6:49

Krupesh Patel

494 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Unable to get the list of topics from AWS MSK using Python

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related