kcat

kcat is a generic non-JVM producer and consumer for Apache Kafka >=0.8, think of it as a netcat for Kafka. You can read more about kcat here https://github.com/edenhill/kcat

We use SASL SCRAM for authentication for our Apache Kafka cluster, below you can find an example for both consuming and producing messages.

macOS

For macOS kcat comes pre-built with SASL_SSL support and can be installed with brew install kcat.

Ubuntu/Debian

You have to compile kcat in order to get SASL_SSL support.

sudo apt-get install kafkacat

Verify that security.protocol includes sasl_ssl and sasl.mechanisms includes SCRAM-SHA-512

./kcat -X list | grep sasl
builtin.features                         |  *  |                 | gzip, snappy, ssl, sasl, regex, lz4, sasl_plain, sasl_scram, plugins | Indicates the builtin features for this build of librdkafka. An application can either query this value or attempt to set it with its list of required features to check for library support. 
*Type: CSV flags*
security.protocol                        |  *  | plaintext, ssl, sasl_plaintext, sasl_ssl |     plaintext | Protocol used to communicate with brokers. 
*Type: enum value*
sasl.mechanisms                          |  *  |                 |        GSSAPI | SASL mechanism to use for authentication. Supported: GSSAPI, PLAIN, SCRAM-SHA-256, SCRAM-SHA-512. **NOTE**: Despite the name only one mechanism must be configured. 
*Type: string*

Metadata

Printing the metadata of the cluster will show you the replicas and in-sync-replicas for each partition as well as which brokers are online which can be helpful at times.

  kcat -b rocket.srvs.cloudkafka.com:9094 \
    -X security.protocol=SASL_SSL -X sasl.mechanisms=SCRAM-SHA-512 \
    -X sasl.username=USERNAME -X sasl.password=PASSWORD -L

Consumer

  kcat -b rocket.srvs.cloudkafka.com:9094 \
    -X security.protocol=SASL_SSL -X sasl.mechanisms=SCRAM-SHA-512 \
    -X sasl.username=USERNAME -X sasl.password=PASSWORD -C TOPIC

Producer

The producer reads data from stdin so for this example we just pipe in a string

echo "Hello from kcat" | kcat -b rocket.srvs.cloudkafka.com:9094 \
  -X security.protocol=SASL_SSL -X sasl.mechanisms=SCRAM-SHA-512 \
  -X sasl.username=USERNAME -X sasl.password=PASSWORD -P -t TOPIC

Debugging

Kcat is a great tool for debugging, you can give it some more arguments and it will print out everything you need to know on why something is wrong. Both the consumer and the producer can print out debug messages. Run the same commands as above but add -v -X debug=generic,broker,security

Kcat with SSL

Kcat supports all of available authentication mechanisms in Kafka, one popular way of authentication is using SSL.

To use SSL authentication with Kcat you need to provide a private key, a signed certificate.

Example, listing kafka metadata:

kcat -b test-speedcar-01.srvs.cloudkafka.com:9093 \
-X security.protocol=SSL -X ssl.key.location=private_key.pem -X ssl.key.password=my_key_password \
-X ssl.certificate.location=signed_cert.pem.txt \
-X ssl.ca.location=ca_cert.pem -L

In the above example the broker is a hosted broker here at CloudKarafka, so we supply an extra argument where we can specify the CA certificate. This is used to verify the brokers key, it might not be needed if you host the broker internally or locally on your computer.

CloudKarafka and SSL

We recommend our customers to use SASL/SCRAM as authentication mechanism but sometimes this isn't an option so we support SSL as well. SSL based authentication is only available on our dedicated plans, for the shared plan only SASL/SCRAM is available. You can find instructions on how to generate the private key, and the signed cert, under the menu option Certificates.