Introduction to Kafka infrastructure, workflow, and storage mechanism

One, Kafka infrastructure

• Producer: The message producer, the client that sends messages to the Kafka broker;
• Consumer: message consumer, the client that fetches messages from the kafka broker;
• Consumer Group: Consumer group, composed of multiple consumers. Each consumer in a consumer group is responsible for consuming data in different partitions. A partition can only be consumed by consumers in one group; consumer groups do not affect each other. All consumers belong to a certain consumer group, that is, the consumer group is a logical subscriber .
• Broker: A kafka server is a broker. A cluster is composed of multiple brokers. A broker can hold multiple topics.
• Topic: It can be understood as a queue, and both the producer and the consumer are facing a topic;
• Partition: In order to achieve scalability, a very large topic can be distributed to multiple brokers (ie servers), a topic can be divided into multiple partitions, and each partition is an ordered queue;
• Replica: Replica. To ensure that when a node in the cluster fails, the partition data on the node is not lost, and Kafka can still continue to work. Kafka provides a copy mechanism. Each partition of a topic has several copies. A leader and several followers.
• leader: The "master" of multiple copies of each partition, the objects that producers send data to, and the objects that consumers consume data are all leaders.
• Follower: The "slave" in multiple copies of each partition synchronizes data from the leader in real time and keeps the data synchronized with the leader. When the leader fails, a follower will become a new follower.

Two, Kafka workflow and file storage mechanism

1. Kafka workflow

In Kafka, messages are classified by topic. Producers produce messages and consumers consume messages, which are all topic-oriented
. Topic is a logical concept, and partition is a physical concept. Each partition corresponds to a log file
, and the log file stores the data produced by the producer. The data produced by the Producer will be continuously appended to the end of the log file, and each piece of data has its own offset. Each consumer in the consumer group will record in real time which offset they have consumed, so that when the error is restored, they can continue to consume from the last location.

2. Kafka file storage mechanism

2.1, storage mechanism

• Since the messages produced by the producer will continue to be appended to the end of the log file, in order to prevent the log file from being too large and resulting in inefficient data positioning, Kafka adopts a fragmentation and indexing mechanism to divide each partition into multiple segments (logical concept, equal to index +log file).
• Each partition (directory) is equivalent to a huge file that is equally distributed to multiple segment data files of the same size (the number of messages in each segment file is not necessarily equal). This feature is also convenient for the deletion of old segments. That is to facilitate the cleaning of messages that have been consumed, and improve the utilization of disks . Each partition only needs to support sequential read and write . The file life cycle of the segment is determined by the server configuration parameters (log.segment.bytes, log.roll.{ms, hours} and other parameters).
• Each segment corresponds to two files-". index" file and ".log " file. They are represented as segment index files and data files (the purpose of introducing index files is to facilitate the use of binary search to quickly locate the message location). These files are located in a folder, and the naming rule of the folder is: topic name + partition serial number (the first segment of the global partition starts from 0, and the name of each subsequent segment file is named after the offset of the first message of the current segment. , The size of the value is 64 bits, the length of the number is 20 bits, and there is no number filled with 0.). For example, if
  the topic first has three partitions, the corresponding folders are first-0, first-1, and first-2.

00000000000000000000.index
00000000000000000000.log
00000000000000170410.index
00000000000000170410.log
00000000000000239430.index
00000000000000239430.log

[root@centos7-1 data]# pwd
/data/kafka/data
#这个需要安装插件 yum -y install tree
[root@centos7-1 data]# tree
.
├── cleaner-offset-checkpoint
├── meta.properties
├── recovery-point-offset-checkpoint
├── replication-offset-checkpoint
# partition目录(topic名称+分区序号)
├── test-0
# segment索引文件
│   ├── 00000000000000000000.index
# 数据文件
│   ├── 00000000000000000000.log
# 0.8版本之前的kafka没有timeindex文件，这是kafka的具体时间日志
│   └── 00000000000000000000.timeindex
│   ├── 00000000000000170410.index
│   ├── 00000000000000170410.log
│   └── 00000000000000170410.timeindex
├── test-1
│   ├── 00000000000000000000.index
│   ├── 00000000000000000000.log
│   └── 00000000000000000000.timeindex
└── test-2
    ├── 00000000000000000000.index
    ├── 00000000000000000000.log
    └── 00000000000000000000.timeindex

The index and log files are named after the offset of the first message of the current segment. The following figure shows
the structure diagram of index file and log file.

2.2 Detailed explanation of index and log files

The .index index file stores a large amount of index information, the .log data file stores a large amount of message data (Message), and the metadata in the index file points to the physical offset address of the Message in the corresponding data file. Taking the metadata 3,497 in the index file as an example, the third Message (the 368772th message in the global Partition) is indicated in the data file in turn, and the physical offset address of the message is 497. The relationship between the index and the log file is as follows:

2.3, the structure of the message

• CRC32 : 4 bytes, the check code of the message.
• magic : 1 byte, magic number identifier, related to the message format, the value is 0 or 1. When magic is 0, the offset of the message uses absolute offset and there is no timestamp part in the message format; when magic is 1, the offset of the message uses relative offset and there is a timestamp part in the message format. Therefore, the length of the message is different if the magic value is different.
• attributes : 1 byte, the attributes of the message.
  The combination of the 0th and 2nd digits indicates the compression type used by the message, 0 means no compression, 1 means gzip compression, 2 means snappy compression, and 3 means lz4 compression. The third bit indicates the time stamp type, 0 indicates the creation time, and 1 indicates the append time.
• timestamp : Timestamp, its meaning is determined by the third bit of attributes.
• key length : The length of the message key.
• key : The key of the message.
• value length : the value length of the message.
• value : the content of the message

2.4. How to find Message by offset

1. Binary search to obtain the corresponding index index file, and obtain the corresponding physical offset

2. Take the physical offset to the log data file order to find the corresponding message

3. Return the found message

For example, to read the Message with offset=368776, the following two steps are required.

• The first step: Find Segment File.
  00000000000000000000.index represents the first file, the start offset (offset) is 0; the start offset of the second file 00000000000000368770.index is 368770, and so on. The files are named and sorted by the starting offset. As long as the file list is searched according to the offset binary, the specific file can be quickly located.

When offset=368776, locate 00000000000000368770.index|log.

• Step 2: Find Message through Segment File.
  Locate the Segment File through the first step. When offset=368776, locate the metadata physical location of 00000000000000368770.index and the physical offset address of 00000000000000368770.log, and then search through 00000000000000368770.log in order until offset=368776. .
  Segment Index File adopts a sparse index storage method, which can reduce the size of index files, and can directly perform memory operations through the Linux mmap interface. Sparse index sets a metadata pointer for each corresponding Message of the data file. It saves more storage space than dense index, but it takes more time to look up.