table of Contents
partition and consumerGroup and consumer
partition and consumerGroup and consumer
In the same group, the number of consumers cannot be greater than the number of partitions, and the extra messages cannot be received.
The same message can only be consumed by a single consumer in the same group, but it can be consumed by consumers in other groups
broker and cluster and replica
broker: An independent Kafka server is called a broker. The broker receives the message from the producer, sets the offset for the message, and submits the message to disk for storage. The broker provides services to consumers, responds to requests for reading partitions, and returns messages that have been submitted to the disk.
Cluster: Broker nodes handed over to the same zookeeper cluster to manage constitute a Kafka cluster.
The broker is an integral part of the cluster, and each cluster has a broker that also acts as the cluster controller. The controller is responsible for management, including assigning partitions to brokers and monitoring brokers. In a broker, a partition belongs to a broker, and the broker is called the leader of the partition. A partition can be assigned to multiple brokers (when multiple copies of Topic are set), then partition replication will occur. As shown below:
topic and partition
Kafka messages are categorized by topic, which is like a table in a database or a folder in a file system. The topic can be divided into several partitions (Partition), a partition is a commit log. Messages are written to the partition in an append, and then read in a first-in, first-out order. Note that because a topic generally contains several partitions, the order of messages cannot be guaranteed within the entire topic, but the order of messages within a single partition can be guaranteed. The theme is a logical concept. Physically, a theme spans multiple servers.
Note: The number of partitions cannot be reduced, because if the partition is deleted, the data in the partition will also be deleted, resulting in inconsistent data. If you must reduce the number of partitions, you can only delete topic and rebuild)
Offset maintenance
Since the Consumer may experience failures such as power outages and downtime during the consumption process, after the Consumer recovers, it needs to continue to consume from the location before the failure, so the Consumer needs to record where it consumes in real time so that it can continue to consume after the failure is restored. Before Kafka version 0.9, Consumer saves the offset in zookeeper by default. Starting from version 0.9, Consumer saves the offset in a built-in Kafka topic named _consumeroffsets by default. It is unreadable by default and can be read by setting exclude.internal.topics=false in consumer.properties.