1. Get to know kafka
(1) Kafka is a distributed message queue based on the publish/subscribe model.
Two modes of message queue:
- Point-to-point mode (one-to-one): The message producer produces the message and sends it to the queue, and then the message consumer takes it out of the queue and consumes the message. After the message is consumed, there is no longer storage in the queue, so it is impossible for the message consumer to consume the message that has been consumed. Queue supports the existence of multiple consumers, but for a message, only one consumer can consume it.
- Publish/subscribe mode (one-to-many): A message producer (publisher) publishes a message to a topic, and multiple message consumers (subscribers) consume the message at the same time. Unlike the point-to-point method, messages published to a topic will be consumed by all subscribers.
(2) Kafka related concepts:
- Producer: Message producer, which pushes messages to the Broker in the Kafka cluster.
- Consumer: Message consumer, pulls messages from the Kafka cluster and consumes messages.
- Consumer Group: A consumer group consists of one or more Consumers. Each Consumer belongs to a Consumer Group. A consumer group is logically a subscriber. Each consumer in the consumer group is responsible for consuming data from different partitions. A partition can only be consumed by consumers in one group; consumer groups do not affect each other. That is, each message can only be consumed by one Consumer in the Consumer Group; but it can be consumed by multiple Consumer Groups. This achieves unicast and multicast.
- Broker: A Kafka server is a Broker. A cluster consists of multiple Brokers, and each Broker can accommodate multiple Topics.
- Topic: The category or topic of the message, which can be logically understood as a queue. The Producer only pays attention to which Topic the message is pushed to, and the Consumer only pays attention to which Topic it has subscribed to.
- Partition: Considering load balancing and scalability, a Topic can be divided into multiple Partitions, which are physically stored on multiple Brokers in the Kafka cluster. For reliability reasons, each Partition will have a backup Replica.
- Replica: A copy of a Partition. In order to ensure that when a node in the cluster fails, the Partition data on the node will not be lost and Kafka can still continue to work, so Kafka provides a copy mechanism. Each Partition of a Topic has Several copies, one Leader and several Followers.
- Leader: The main role of Replica. Producer and Consumer only interact with Leader.
- Follower: The slave role of Replica, synchronizes data from the Leader in real time, and maintains synchronization with the Leader data. When the Leader fails, a Follower will become the new Leader.
- Controller: One of the servers in the Kafka cluster, used for leader election and various failovers.
- ZooKeeper: Kafka stores meta information of the cluster through Zookeeper.