Send a message
The process is as follows:
- The Producer side directly sends the message to the Leader partition in the Broker
- The Leader partition corresponding to the Broker will first write the message to the Page Cache when it receives the message, and periodically refresh the disk for persistence (sequentially write to the disk)
- The Follower partition pulls the message of the Leader partition and keeps it consistent with the data of the Leader partition. After the message is pulled, it needs to reply an ACK confirmation message to the Leader partition
- After the Leader partition and the Follower partition have synchronized data and received ACKs from all Replica copies in the ISR, the Leader partition will reply the ACK confirmation message to the Producer
consumption news
The process is as follows:
- Consumers need to obtain cluster metadata through the subscription relationship, find the data of the Leader partition corresponding to the relevant Topic, and then actively pull messages from the Kafka cluster through the Pull mode
- After the message is pulled, perform business logic processing. After the processing is completed, ACK confirmation will be performed, that is, the Offset consumption displacement progress record will be submitted.
- Finally, the Offset will be saved to the topic consumer_offsets in the Kafka Broker cluster, and each Consumer will save its own Offset progress
Note: In this process, there is a concept of consumer group. Multiple consumers can form a consumer group, namely Consumer Group, and each consumer group has a GroupId. Consumers in the same Consumer Group can consume data from different partitions under the same Topic, but there will not be multiple Consumers to consume data from the same partition.
Summarize
To sum up, it can be concluded that Kafka has three message delivery processes:
- Producer side sends message to Broker side
- Broker processes messages and persists data
- The Consumer pulls and consumes messages from the Broker