Kafka Concepts
There are a some concepts that is good to be familiar
with before you get started with your first CloudKarafka instance.
-
Kafka cluster
-
In Kafka a cluster is one or more servers called brokers.
-
Message Broker
-
The Kafka server, a queue manager that can handle a large amount of
reads and writes per second from a lot of clients.
Message data is replicated and persisted on the Brokers.
-
Messages
-
Information that is sent from the producer to a consumer through
Kafka. Messages are byte arrays that can store
any object format - strings or JSON as the most common once.
-
Topics
-
Message queues in Kafka are called topics, it is a category or feed
name to which messages are published. Producers write data to
topics and consumers read from topics.
-
Producers and Consumers
-
Producers publish data to the topics of their choice, consumers consumes messages from
the topic. Producers and Consumers can simultaneously write to and read from multiple topics.
-
Distributed
-
Kafka is a distributed system, topics are partitioned and replicated
across multiple nodes.
From wikipedia: "A distributed system is a software system in which components
located on networked computers communicate and coordinate
their actions by passing messages."
-
Partitioned
-
A topic consist of one or more partitions on different brokers in the
cluster. For each topic, the Kafka cluster maintains a partitioned log.
The messages in the partitions are each assigned a sequential id
number called the offset that identifies each message within the partition.
The producer is responsible for choosing which message
to assign to which partition within the topic and the consumer
need to track what messages that have been consumed.
-
Replicated
-
Each partition is replicated across a configurable number of servers
for fault tolerance.