To understand the integration of Zookeeper in Apache Kafka it’s important to understand how the number of nodes in a cluster makes a difference for the Zookeeper function. In this blog post, hopefully, you get an idea.
As mentioned in the previous blog post, Zookeeper is installed and configured by default with CloudKarafka, all depending on the number of nodes in your cluster.
Zookeeper and Majority Quorums
A replicated group of servers in the same application is called a Quorum. ZooKeeper uses Majority Quorums, which means that every voting* requires a majority to vote on.
* Zookeeper uses the notion of Quorum to guarantee a consistent view of the system. For example, when selecting a new leader, manage synchronization of transactions and broadcast of relevant data.
CloudKarafka number of nodes
CloudKarafka offers clusters in sizes of
1, 3, 5 or 7 nodes.
3, 5, or 7 is the most common number of nodes since a higher number of members in a cluster makes the cluster more failure tolerant.
Number of nodes and Zookeeper
1 Node: Of course, you can build your infrastructure with a single node, but you won’t receive the highly reliable and available system as in a cluster with 3 or more nodes.
3 Nodes: An ensemble with 3 nodes will support one failure without loss of service, which is probably fine for most users, and also the most popular setup chosen by customers at CloudKarafka.
5 Nodes (recommended): A five-node cluster allows you to take one server out for maintenance or upgrade and still be able to take a second unexpected failure, without interrupting your service.
7 Nodes (recommended): The same as for 5-node cluster but with the ability to bear the failure of three nodes.
Most of our users doesn’t need to know about Zookeeper presence since its job is to keep the Kafka cluster together. But some clients require a connection to Zookeeper to read information about the Kafka cluster. With CloudKarafka, access to Zookeeper is only allowed over peered VPC networks in the Cloud provider.
In the third part of this blog series on Zookeeper, we will cover Zookeeper Atomic Broadcast (ZAB).
