Kafkacat

kafkacat is a generic non-JVM producer and consumer for Apache Kafka >=0.8, think of it as a netcat for Kafka. You can read more about kafkacat here https://github.com/edenhill/kafkacat

We use SASL SCRAM for authentication for our Apache Kafka cluster, below you can find an example for both consuming and producing messages.

macOS

For macOS kafkacat comes pre-built with SASL_SSL support and can be installed with brew install kafkacat.

Ubuntu/Debian

You have to compile kafkacat in order to get SASL_SSL support.

sudo apt-get install build-essential python libssl-dev openssl
git clone https://github.com/edenhill/kafkacat.git
cd kafkacat
./bootstrap.sh

Verify that security.protocol includes sasl_ssl and sasl.mechanisms includes SCRAM-SHA-256

./kafkacat -X list | grep sasl
builtin.features                         |  *  |                 | gzip, snappy, ssl, sasl, regex, lz4, sasl_plain, sasl_scram, plugins | Indicates the builtin features for this build of librdkafka. An application can either query this value or attempt to set it with its list of required features to check for library support. <br>*Type: CSV flags*
security.protocol                        |  *  | plaintext, ssl, sasl_plaintext, sasl_ssl |     plaintext | Protocol used to communicate with brokers. <br>*Type: enum value*
sasl.mechanisms                          |  *  |                 |        GSSAPI | SASL mechanism to use for authentication. Supported: GSSAPI, PLAIN, SCRAM-SHA-256, SCRAM-SHA-512. **NOTE**: Despite the name only one mechanism must be configured. <br>*Type: string*

Consumer

  export BROKERS=<broker1>:9094,<broker2>:9094,<broker3>:9094
  export USERNAME=<username>
  export PASSWORD=<password>
  export TOPIC=$USERNAME-default
  kafkacat -b $BROKERS -C -X security.protocol=SASL_SSL -X sasl.mechanisms=SCRAM-SHA-256 -X sasl.username=$USERNAME -X sasl.password=$PASSWORD -p 1 -t $TOPIC

Producer

The producer reads data from stdin so for this example we just pipe in a string

  export BROKERS=<broker1>:9094,<broker2>:9094,<broker3>:9094
  export USERNAME=<username>
  export PASSWORD=<password>
  export TOPIC=$USERNAME-default
  echo "Hello kafkacat!" | kafkacat -b $BROKERS -P -X security.protocol=SASL_SSL -X sasl.mechanisms=SCRAM-SHA-256 -X sasl.username=$USERNAME -X sasl.password=$PASSWORD -t $TOPIC

Debugging

Kafkacat is a great tool for debugging, you can give it some more arguments and it will print out everything you need to know on why something is wrong. Both the consumer and the producer can print out debug messages. Run the same commands as above but add -v -X debug=generic,broker,security

Kafkacat with SSL

Kafkacat supports all of available authentication mechanisms in Kafka, one popular way of authentication is using SSL.

To use SSL authentication with Kafkacat you need to provide a private key, a signed certificate.

Example, listing kafka metadata:

kafkacat -b test-speedcar-01.srvs.cloudkafka.com:9093 \ 
-X security.protocol=SSL -X ssl.key.location=private_key.pem -X ssl.key.password=my_key_password \
-X ssl.certificate.location=signed_cert.pem.txt \
-X ssl.ca.location=ca_cert.pem -L

In the above example the broker is a hosted broker here at CloudKarafka, so we supply an extra argument where we can specify the CA certificate. This is used to verify the brokers key, it might not be needed if you host the broker internally or locally on your computer.

CloudKarafka and SSL

We recommend our customers to use SASL/SCRAM as authentication mechanism but sometimes this isn't an option so we support SSL as well.

SSL based authentication is only available on our dedicated plans, for the shared plan only SASL/SCRAM is available.

You can find instructions on how to generate the private key, and the signed cert, under the menu option Certificates.