There are two very good client for Apache Kafka written for Python.
One is https://github.com/mumrah/kafka-python which can be installed with
pip install kafka-python
And then we have the client written by Confluent, the developers of Kafka, https://github.com/confluentinc/confluent-kafka-python which like all their clients are a wrapper around the excellent librdkafka
We recommend to use the client from Confluent since it supports authentication with SASL SCRAM which is what we use at CloudKaraka.
To get started, you first need all the connection variables, which you can get from the provider you used to create the CloudKarafka instance.
c = Consumer(**conf) c.subscribe(topics) while True: msg = c.poll(timeout=1.0) if msg.error(): print(msg.error()) else: sys.stderr.write('%% %s [%d] at offset %d with key %s:\n' % (msg.topic(), msg.partition(), msg.offset(), str(msg.key()))) print(msg.value())
The complete code example, with configuration, can be found here: https://github.com/CloudKarafka/python-kafka-example