Kafka is everywhere in the field of big data messaging middleware, now widely used within the enterprise real-time data pipeline, and to help companies build their own stream computing applications. While Kafka is doing disk-based data storage, but it has the characteristics of high performance, high throughput, low latency, throughput often tens of thousands, tens of millions, of which reason is worth a closer look. This article is part Kafka literacy series, let us grasp Kafka various sophisticated design.
Sequential read and write
Kafka is well known that logs messages persisted to local disk, most people would think that poor performance disk read and write, how to ensure the performance of Kafka might be questioned. In fact, whether or disk memory, fast or slow key lies in addressing mode, the disk is divided into sequential read and write and random read and write, sequential read and write into the same memory and random read and write. Based on random disk read and write really slow, but the order of disk read and write performance is very high, generally higher than the disk random read and write to three orders of magnitude, disk sequential read and write performance in some cases even higher than the random read memory write, here is a comparison chart on the performance of leading academic journals ACM Queue:
Sequential disk read and write disk usage patterns of the most regular and Caozuojitong also done a lot of optimization of this model, Kafka is the use of sequential read and write disk to improve performance. Kafka's message is continually appended to the end of the local disk file, rather than random writes, which makes Kafka write throughput has been significantly improved.
page Cache
In order to optimize read and write performance, Kafka advantage of the operating system itself Page Cache, is to use the operating system's own memory rather than the JVM memory space. Benefits of doing so are:
- Avoid consumption of Object: If you are using Java heap memory consumption of Java objects larger, often twice or even more of the stored data.
- GC avoid problems: With data growing JVM garbage collection will become complicated and slow, use the system cache GC problem would not exist.
Compared to the use of the JVM or in-memory cache and other data structures, using the operating system's Page Cache is more simple and reliable. First, the operating system level cache utilization will be higher, since they are stored in a compact byte structure rather than a separate object. Second, the operating system itself has done a lot for Page Cache optimized to provide a variety of mechanisms write-behind, read-ahead and flush and so on. Furthermore, even if the service restart the process, the system cache still does not disappear, avoid the in-process cache rebuild cache process.
Page Cache by the operating system, read and write operations are substantially Kafka been greatly improved memory-based, read and write speed.
Zero-copy
Here is mainly about Kafka use linux operating system's "zero-copy (zero-copy)" in the consumer end of the mechanism to do optimization. First, to understand the data file transmitted from the transmission path to a conventional socket network connections:
- Operating system reads data from disk to kernel space (kernel space) of Page Cache
- Page Cache application reads data to user space (user space) buffer
- The application user space buffer data is written back to the kernel space socket buffer (socket buffer)
- The operating system copies the data from the socket buffer is sent to the network buffer NIC
The process involves four copy operations and the context switching system 2, the performance is actually very inefficient. linux operating system, "zero copy" mechanism uses sendfile method, allows the operating system to send data from the Page Cache directly to the network, only the last step of the copy operation to copy data to the NIC buffer, thus avoiding re-copying the data. Diagram is as follows:
In this "zero copy" mechanism, Page Cache combine sendfile method, the performance of Kafka consumer side has increased dramatically. This is why sometimes the consumer side in the continued consumption data, we do not see the disk io is relatively high, at the moment it is cached in the operating system to provide data.
Partition segments
Kafka's message is classified and stored by topic, topic data is in accordance with a partition to a partition that is stored to a different broker node. Each partition corresponds to a file folder on the operating system, partition segment according to the segment is actually stored. It is also very consistent with the design idea of distributed system partition divided barrel.
By this design of partition segment, Kafka's message segment in a message is actually a small segment of each file operations are directly distributed storage operation. For further query optimization, Kafka has established a default index file for the data file segments is .index files on the file system. This partitioning + index segment design, not only improving the efficiency of data reading, but also improve the operation of data parallelism.
to sum up
In summary, using sequential read Kafka, Page Cache, and zero copy partition segments like these designs, together with optimization done in terms of an index, data reading and writing is additionally Kafka batch rather than a single strip, having such a high Kafka performance, high-throughput, low-latency characteristics. In this way, Kafka provide large-capacity disk storage has become an advantage. As the school talent coarse shallow, local representation incorrect advice welcome.