table of Contents
1. Overview of storage structure
- Each partion (folder) is equivalent to a huge file that is evenly distributed to multiple data files of equal size segment (segment). However, the number of messages in each segment file is not necessarily equal. This feature facilitates the high-speed deletion of old segment files. (By default, the size of each file is 1G)
- Each partiton only needs to support sequential reading and writing. The segment file life cycle is determined by the server configuration parameters.
1. Segment file storage structure in partiton
Segment file composition: consists of 2 parts. They are index file and data file respectively. These two files correspond one by one and appear in pairs. The suffixes ”.index”and suffixes “.log”respectively represent segmentindex file and data file .
Segment file naming rules: The first segment of the global partition starts from 0, and the name of each segment file may be the offset value of the last message of the previous segment file.
The maximum value is a 64-bit long. The length of 19 digits is filled with zeros.
itcast@Server-node:/mnt/d/kafka_2.12-2.2.1$ ll /tmp/kafka/log/heima-0/
total 20480
drwxr-xr-x 1 itcast sudo 512 Aug 29 09:38 ./
drwxrwxrwx 1 dayuan dayuan 512 Aug 29 09:41 ../
-rw-r--r-- 1 itcast sudo 10485760 Aug 29 09:38 00000000000000000000.index
-rw-r--r-- 1 itcast sudo 0 Aug 29 09:38 00000000000000000000.log
-rw-r--r-- 1 itcast sudo 10485756 Aug 29 09:38 00000000000000000000.timeindex
-rw-r--r-- 1 itcast sudo 8 Aug 29 09:38 leader-epoch-checkpoint
itcast@Server-node:/mnt/d/kafka_2.12-2.2.1$
Two, log index
1. Segmentation of data files
One of Kafka's methods to solve query efficiency is to segment data files. For example, there are 100 messages, and their offset is from 0 to 99. Suppose the data file is divided into 5 segments, the first segment is 0-19, the second segment is 20-39, and so on, each segment is placed in a separate data file, and the data file is named after the smallest offset in the segment. In this way, when searching for a Message with a specified offset, binary search can be used to locate which segment the Message is in.
2. Offset index
Data file segmentation makes it possible to find the Message corresponding to the offset in a smaller data file, but this still requires sequential scanning to find the Message corresponding to the offset. In order to further improve the search efficiency, Kafka creates an index file for each segmented data file. The file name is the same as the data file name, but the file extension is .index.
For example: to find Message with absolute offset of 7:
The first is to use binary search to determine which LogSegment it is in, which is naturally in the first Segment. Open the index file of this segment, and use binary search to find the largest offset in the index entries whose offset is less than or equal to the specified offset. Naturally, the index with offset 6 is what we are looking for. Through the index file, we know that the position of Message with offset 6 in the data file is 9807.
Open the data file and scan sequentially from the place at 9807 until the message with offset 7 is found.
This set of mechanisms is based on the orderly offset. The index file is mapped to memory, so the search speed is still very fast.
Word, Kafka uses the Message storage partition (Partition) , the segment (logsegment) and sparse index these means to achieve a high efficiency.
Three, log cleanup
1. Log deletion
Kafka log manager allows custom deletion strategies . The current strategy is to delete the logs whose modification time is N days ago (delete by time). Another strategy can also be used: the strategy of retaining the last NGB data (delete by size). In order to avoid blocking the read operation during deletion, a copy-on-write implementation is adopted . When the deletion operation is performed, the binary search function of the read operation is actually performed on a static snapshot copy, which is similar to Java CopyOnWriteArrayList. Kafka consumption log deletion thought: Kafka divides a large partition file in a topic into multiple small file segments. Through multiple small file segments, it is easy to periodically clear or delete files that have been consumed to reduce disk usage .
log.cleanup.policy=delete 启用删除策略
直接删除，删除后的消息不可恢复。可配置以下两个策略：
清理超过指定时间清理：
log.retention.hours=16
超过指定大小后，删除旧的消息：
log.retention.bytes=1073741824
2. Log compression
Compress the data and only keep the data of the last version of each key. First, set the log.cleaner.enable=trueenable cleaner in the configuration of the broker . This is disabled by default. Set the log.cleanup.policy=compactenable compression strategy in the Topic configuration .
The compressed offset may be discontinuous. For example, there are no 5 and 7 in the above figure, because the messages of these offsets have been merged. When messages are consumed from these offsets, the messages corresponding to the offsets larger than this offset will be obtained. For example, when trying to get a message with an offset of 5, you will actually get a message with an offset of 6, and start consumption from this position.
This strategy is only suitable for special scenarios. For example, the key of the message is the user ID, and the message body is the user's information. Through this compression strategy, the latest information of all users is saved in the entire message set.
The compression strategy supports deletion. When the latest version of a key has no content, the key will be deleted, which also conforms to the above logic.
Fourth, the advantages of disk storage
When Kafka was designed, a file append method was used to write messages, that is, new messages can only be appended to the end of the log file, and messages that have been written are not allowed to be modified. This method is a typical sequential write to this Judgment operation, so even if Kafka uses disk as a storage medium, the throughput that can be achieved is very impressive.
Kafka uses a large amount of page cache , which is also one of the important factors for Kafka to achieve high throughput .
In addition to message sequence addition , page caching and other technologies, Kafka also uses zero-copy technology to further improve performance. " Zero copy technology " only needs to copy the data of the disk file to the page cache once, and then send the data from the page cache directly to the network (when sending to different subscribers, the same page cache can be used), avoiding Repeat the copy operation. If there are 10 consumers, under the traditional method, the number of data copying times is 4*10=40 times, while the use of "zero copy technology" only needs 1+10=11 times, once is copied from disk to page cache, 10 times means 10 consumers each read the page cache once.
to sum up
This chapter mainly describes the knowledge points related to storage in Kafka, including Kafka's own log format , log index , log cleanup, etc., and also involves knowledge of the underlying physical storage. Through the study of this chapter, you can have a deeper understanding of the core mechanism of Kafka.
Kafka partition management (priority copy election, partition redistribution) interested students can click on the link to read my last article, which also contains the method of obtaining this article!