Features of Kafka:
- High throughput and low latency: Kafka can process hundreds of thousands of messages per second, and its latency is as low as a few milliseconds. Each topic can be divided into multiple partitions, and consumer groups can consume partitions.
- Scalability: Kafka cluster supports hot expansion
- Persistence and reliability: messages are persisted to the local disk, and data backup is supported to prevent data loss
- Fault tolerance: Allow nodes in the cluster to fail (if the number of copies is n, then n-1 nodes are allowed to fail)
- High concurrency: support thousands of clients to read and write at the same time
Consumer Group：
Consumer group, each consumer in the consumer group is responsible for consuming data in different partitions to improve consumption power. Logically a subscriber.
Topic：
It can be understood that in a queue, Topic categorizes consumption, and producers and consumers are facing the same Topic.
Partition：
In order to achieve scalability and improve concurrency, a Topic is distributed to multiple Brokers in the form of multiple Partitions, and each Partition
is an ordered queue. Each Partition of a topic has several replicas, one leader and several
followers. The objects to which producers send data and the objects to which consumers consume data are leaders. Follower is responsible for
synchronizing data from the Leader in real time and keeping the data synchronized with the Leader. When the leader fails, a follower will become the new leader.
Offset：
The location information of the consumer's consumption, monitor where the data is consumed, and when the consumer hangs up and resumes, they can continue to consume from the consumer location.
Zookeeper：
The Kafka cluster can work normally and needs to rely on Zookeeper. Zookeeper helps Kafka store and manage cluster information. Broker
and Topic register to Zookeeper and generate temporary nodes of Znode.
The relationship between Broker, Topic, Offset, and Partition, including the association relationship between Consumer, is stored in zk. The election mode of Leader node also uses Zookeeper
's election mechanism.