Chapter 2 Kafka Quick Guide
Kafka related terms
Broker- Kafka cluster contains one or more servers, this kind of server is called broker;
Topic -every message published to Kafka cluster has a category, this category is called topic. (Physically, different Topic messages are stored separately. Although logically one Topic message is stored on one or more brokers, users only need to specify the Topic of the message to produce or consume data without having to care where the data is stored)
Partition — —Partitioning is a physical concept. Each Topic contains one or more Partitions.
Producer — responsible for publishing messages to Kafka broker
Consumer — message consumer, client that reads messages from Kafka broker.
Consumer Group -Each Consumer belongs to a specific Consumer Group (you can specify the group name for each Consumer, if you do not specify the group name, it belongs to the default group)
The working mode of Kafka is that the producer appends a message to the topic. The topic stores data in the form of a log. The log is divided into different partitions on the physical structure and on different brokers in the cluster. Consumers subscribe to a topic and read messages from the topic's log.
Kafka uses ZooKeeper to manage the cluster. ZooKeeper is used to coordinate server or cluster topology. ZooKeeper is a consistent file system for configuration information.
Understanding of the working mode of Kafka cluster
The kafka cluster consists of a proxy server, that is, a broker, and a zookeeper management component.
When a topic is created, its messages store data in the form of logs. The log will be divided into various partitions and distributed on each broker. The logs will be backed up on each broker at the same time, and the backup mechanism maintains fault tolerance. When a backup mechanism exists, each partition has multiple storage copies on different brokers. Each partition will define a leader broker and other follower brokers. This allocation is done by the agent controller. I think this controller is available for each broker, because its role is to specify the leader for the partition and reallocate the follower or leader when the follower broker fails. ZooKeeper is a component of the management cluster, it is not a broker, it is responsible for monitoring the status of each broker. If a broker is not available, ZooKeeper is responsible for removing it from the cluster, and it records which server the leader of each topic is. It also performs access control. [There is a doubt here, my understanding is that each partition of each topic has a leader broker, but the article states that there is a leader broker for each topic? 】
