kafkacat is a generic non-JVM producer and consumer for Apache Kafka >=0.8, think of it as a netcat for Kafka. You can read more about kafkacat here https://github.com/edenhill/kafkacat
We use SASL SCRAM for authentication for our Apache Kafka cluster, below you can find an example for both consuming and producing messages.
For macOS kafkacat comes pre-built with SASL_SSL support and can be installed with
brew install kafkacat.
You have to compile kafkacat in order to get SASL_SSL support.
sudo apt-get install build-essential python libssl-dev openssl git clone https://github.com/edenhill/kafkacat.git cd kafkacat ./bootstrap.sh
Verify that security.protocol includes sasl_ssl and sasl.mechanisms includes SCRAM-SHA-256
./kafkacat -X list | grep sasl builtin.features | * | | gzip, snappy, ssl, sasl, regex, lz4, sasl_plain, sasl_scram, plugins | Indicates the builtin features for this build of librdkafka. An application can either query this value or attempt to set it with its list of required features to check for library support.
*Type: CSV flags* security.protocol | * | plaintext, ssl, sasl_plaintext, sasl_ssl | plaintext | Protocol used to communicate with brokers.
*Type: enum value* sasl.mechanisms | * | | GSSAPI | SASL mechanism to use for authentication. Supported: GSSAPI, PLAIN, SCRAM-SHA-256, SCRAM-SHA-512. **NOTE**: Despite the name only one mechanism must be configured.
export BROKERS=broker1:9094,broker2:9094,broker3:9094 export USERNAME=username export PASSWORD=password export TOPIC=$USERNAME-default kafkacat -b $BROKERS -C -X security.protocol=SASL_SSL -X sasl.mechanisms=SCRAM-SHA-256 -X sasl.username=$USERNAME -X sasl.password=$PASSWORD -p 1 -t $TOPIC
The producer reads data from stdin so for this example we just pipe in a string
export BROKERS=broker1:9094,broker2:9094,broker3:9094 export USERNAME=username export PASSWORD=password export TOPIC=$USERNAME-default echo "Hello kafkacat!" | kafkacat -b $BROKERS -P -X security.protocol=SASL_SSL -X sasl.mechanisms=SCRAM-SHA-256 -X sasl.username=$USERNAME -X sasl.password=$PASSWORD -t $TOPIC
Kafkacat is a great tool for debugging, you can give it some more arguments and it will print out everything you need to know on why something is wrong. Both the consumer and the producer can print out debug messages. Run the same commands as above but add
-v -X debug=generic,broker,security
Kafkacat supports all of available authentication mechanisms in Kafka, one popular way of authentication is using SSL.
To use SSL authentication with Kafkacat you need to provide a private key, a signed certificate.
Example, listing kafka metadata:
kafkacat -b test-speedcar-01.srvs.cloudkafka.com:9093 \ -X security.protocol=SSL -X ssl.key.location=private_key.pem -X ssl.key.password=my_key_password \ -X ssl.certificate.location=signed_cert.pem.txt \ -X ssl.ca.location=ca_cert.pem -L
In the above example the broker is a hosted broker here at CloudKarafka, so we supply an extra argument where we can specify the CA certificate. This is used to verify the brokers key, it might not be needed if you host the broker internally or locally on your computer.
We recommend our customers to use SASL/SCRAM as authentication mechanism but sometimes this isn't an option so we support SSL as well.
SSL based authentication is only available on our dedicated plans, for the shared plan only SASL/SCRAM is available.
You can find instructions on how to generate the private key, and the signed cert, under the menu option Certificates.