FAQ: How to purge a Kafka topic

Safe, solid systems are often built and configured to minimize the risk of losing data. On the contrary, sometimes you might want to remove data from your system. In this article, you will learn how this is done in Apache Kafka by showing how to purge a Kafka topic.

You created a topic, and you populated it, right? Bravo! But now you want to remove the data from it for whatever reason.

You may have your reasons, but I must warn you that this action is destructive and irreversible. If you want to do that in a production environment, think twice and make sure you consider all the consequences and really want it to happen.

While a common way to remove data is simply to delete it and recreate the topic, that is not the best practice when the goal is simply to remove the data.

To empty all the data from the topic, it is enough to set the data retention time to zero.

Before doing that, it's best to check if there is any pre-existing retention policy specifically set on the target topic:

kafka-topics.sh --bootstrap-server broker_IP_or_host:9092 --describe --topic target_topic

You will see something like:

Topic: target_topic     PartitionCount: 1    ReplicationFactor: 1    Configs: ....

If retention.ms is mentioned in the 'Configs' section, take note of it as you will need it later. Then we can then proceed to purge all the data from the topic:

kafka-configs.sh --bootstrap-server broker_IP_or_host:9092 --alter --entity-type topics --entity-name target_topic --add-config retention.ms=0

Kafka will then proceed to purge all the data from the topic. When the purging is done, you can then restore the old settings (if existing):

kafka-configs.sh --bootstrap-server broker_IP_or_host:9092 --alter --entity-type topics --entity-name target_topic --add-config retention.ms=old_setting

Or, if retention was not set on the target topic, just delete the topic configuration you introduced to purge the topic:

kafka-configs.sh --bootstrap-server broker_IP_or_host:9092 --alter --entity-type topics --entity-name target_topic --delete-config retention.ms

We hope that you found this information useful and that you will use this knowledge to purge your Kafka topics carefully. If you have any questions about this article or related topics, feel free to contact us at contact@cloudkarafka.com .

All the best

CloudKarafka team