When you are about to set up and run Kafka, the most important question to ask is what you want to achieve? In many businesses, the goal is to create a high performance and reliable setup in which the messages flowing within are both fast and in the correct order. That's why we sometimes receive the question regarding the number of partitions and the possibility to keep a strict ordering of the messages, even when you are using more than one?
The question is accurate since most businesses require things to be in the correct order, for example, businesses keeping track of user actions, metrics and user checkout flow etc.
The simple answer to this particular question is: Yes. If you want a strict order of all messages going to one topic, then you must use only one partition. However, a higher number of partitions is preferable for high throughput in Kafka, and these two factors can seem incompatible with each other.
But , in most cases, we have seen that people rather than the above statement, would prefer messages to be ordered based on a certain property in the message. This means that you, can use multiple partitions and still reach the result you wish for, i.e a strict order of messages while using numerous partitions.
For example:
If you have a topic with user actions which must be ordered, but only ordered per user, the need for using only one partition is no longer needed. This action can be supported by having multiple partitions but using a consistent message key, for example, user id. This will guarantee that all messages for a certain user always ends up in the same partition and thus is ordered.
A consumer will consume from one or more partition, but you will never have two consumers consuming from one partition. So in the case above, if you have 10 partitions and you use 10 consumers, one consumer will get all messages related to the same user and thus be processed in order.
So the answer is as simple as this:
If all messages must be ordered within one topic, use one partition, but if messages can be ordered per a certain property, set a consistent message key and use multiple partitions.
This way you can keep your messages in strict order and keep high Kafka throughput.
And as always, feel free to send us any questions or feedback you might have at support@cloudkarafka.com
