Kafka Concepts

There are a some concepts that is good to be familiar with before you get started with your first CloudKarafka instance.

Kafka cluster

In Kafka a cluster is one or more servers called brokers.

Message Broker

The Kafka server, a queue manager that can handle a large amount of reads and writes per second from a lot of clients. Message data is replicated and persisted on the Brokers.

Messages

Information that is sent from the producer to a consumer through Kafka. Messages are byte arrays that can store any object format - strings or JSON as the most common once.

Topics

Message queues in Kafka are called topics, it is a category or feed name to which messages are published. Producers write data to topics and consumers read from topics.

Producers and Consumers

Producers publish data to the topics of their choice, consumers consumes messages from the topic. Producers and Consumers can simultaneously write to and read from multiple topics.

Distributed

Kafka is a distributed system, topics are partitioned and replicated across multiple nodes. From wikipedia: "A distributed system is a software system in which components located on networked computers communicate and coordinate their actions by passing messages."

Partitioned

A topic consist of one or more partitions on different brokers in the cluster. For each topic, the Kafka cluster maintains a partitioned log. The messages in the partitions are each assigned a sequential id number called the offset that identifies each message within the partition. The producer is responsible for choosing which message to assign to which partition within the topic and the consumer need to track what messages that have been consumed.

Replicated

Each partition is replicated across a configurable number of servers for fault tolerance.