index
Sparse memory, to establish an index (such purpose is to reduce the size of the index file) every predetermined bytes of data.
The following figure shows a partition index schematic:
Note :
- Now 6 and 8 established the index, if you are looking for 7, 8 will first find and then, to find a post-86 index, and then do the dichotomy between the two indexes, to find the location of 7 2
- Each log file is divided into multiple segment
By calling kafka own tools, you can see data in the log
> bin/kafka-run-class.sh kafka.tools.DumpLogSegments --files /root/kafka/kafka-logs/streams-plaintext-input-0/00000000000000000000.log --print-data-log --verify-index-only
kafka log into index and log, two pairs; index file storage metadata (data that describes data, which is probably why such a large index file of the reason), log store messages. The index pointing to the corresponding log file metadata file migration address message; e.g. 2,128-finger second data log file at offset 128; and the physical addresses (specified in the index file) may be positioned to offset address + message.
Because each message is append to the partition in order to write a disk, so efficiency is very high (proven, sequential write disk memory efficiency is even higher than the random write, this is a very important guarantee for high-throughput Kafka) .
