Opening
In the first knowledge of kafka, I talked about the benefits of using MQ (message queue) to design the system: business decoupling, traffic peak reduction, and flexible expansion
There are many popular MQs, because our company chose to use Kafka in the technology selection, so I have compiled an introductory knowledge about Kafka. Through technology selection, we have compared the mainstream MQ in the industry. The biggest advantage of Kakfa is its high throughput .
Kafka is a high-throughput, low-latency, high-concurrency, high-performance messaging middleware, which is extremely widely used in the field of big data. A well-configured Kafka cluster can even achieve ultra-high concurrent writes of hundreds of thousands or millions of writes per second.
So how does Kafka achieve such high throughput and performance? After getting started, let’s take a closer look at Kafka’s architecture design principles. Mastering these principles will have an advantage in Internet interviews.
Endurance
Kafka relies on the file system for message storage and caching. Every time data is received, it will be written to the disk. The general impression of "disk speed" makes people doubt that the persistence architecture can provide strong performance.
In fact, the speed of disks is much slower and faster than people expected, depending on how people use disks. And a properly designed disk structure can usually be as fast as a network.
Through the comparison of the above figure, we can see that sequential disk access is actually faster than random memory access in some cases. In fact, Kafka uses this advantage to achieve high-performance disk writes.
Page caching technology + disk sequential write
In order to ensure disk write performance, Kafka first implements file writing based on the page cache of the operating system.
The operating system itself has a layer of cache, called page cache, which is a cache in memory. We can also call it os cache, which means the cache managed by the operating system itself.
When you write a disk file, you can write it directly to the os cache, that is, just write it to the memory. Then the operating system decides when to actually flush the data in the os cache to the disk.
Through the above figure, the write performance of the disk file can be improved a lot. In fact, this method is equivalent to writing to the memory, not writing to the disk.
Sequential write to disk
In addition, there is a very key point. Kafka writes data in a disk sequential manner when writing data, that is, only appends the data to the end of the file (append), rather than at a random location in the file. change the data.
For ordinary mechanical hard disks, if you write randomly, the performance is indeed extremely low. This involves the problem of disk addressing. But if you just append the end of the file to write data in a sequential manner, then the performance of this disk sequential write can basically be about the same as the performance of writing to the memory itself.
To summarize: Kafka is based on page caching technology + disk sequential write technology to achieve ultra-high performance of writing data.
Therefore, to ensure that the core point of writing tens of thousands or even hundreds of thousands of data per second is to improve the performance of each data writing as much as possible, so that more data can be written per unit time and throughput can be improved. .
Zero-copy technology (zero-copy)
After finishing writing this piece, let's talk about consumption.
Everyone should know that we often consume data from Kafka, so when we consume data, we actually read a piece of data from Kafka's disk file and send it to downstream consumers, as shown in the following figure:
If Kafka reads data from the disk and sends it to downstream consumers in the above manner, the approximate process is:
- First see if the data to be read is in the os cache, if not, read the data from the disk file and put it into the os cache
- Then copy the data from the OS cache of the operating system to the cache of the application process, then copy the data from the cache of the application process to the Socket cache at the operating system level, and finally extract the data from the Soket cache and send it to the network card. Finally Send out to downstream consumers
The whole process is as follows:
As can be seen from the above figure, there are two unnecessary copies of this whole process.
One is from the cache of the operating system to the cache of the application process, and then from the application cache back to the Socket cache of the operating system.
And in order to perform these two copies, several context switches occurred in the middle. For a while, the application was executing, and the context was switched to the operating system for execution.
Therefore, reading data in this way consumes performance.
In order to solve this problem, Kafka introduces zero-copy technology when reading data.
In other words, the data in the cache of the operating system is directly sent to the network card and then transmitted to downstream consumers. The step of copying data is skipped twice in the middle. Only one descriptor will be copied in the Socket cache, and it will not be copied. The data is stored in the Socket cache.
Taste this exquisite process it
by zero-copy technology, there is no need to copy os cache in the data cache to the application, and then copy the application from cache to cache Socket, two copies are omitted, so called zero-copy.
Socket cache is just the descriptor of the copy data, and then the data is sent directly from the os cache to the network card. This process greatly improves the performance of reading file data during data consumption.
And everyone will notice that when reading data from the disk, it will first check whether there is any in the os cache memory. If so, the data is actually read directly from the memory.
If the Kafka cluster is well tuned, you will find that a large amount of data is written directly to the os cache, and then when the data is read, it is also read from the os cache.
It is equivalent to Kafka providing data writing and reading completely based on memory, so the overall performance will be extremely high.
to sum up
By learning the excellent design of Kafka, we understand the use of Kafka's underlying page caching technology, the idea of disk sequential writing, and the use of zero-copy technology, so that Kafka can have such high performance and achieve a throughput of hundreds of thousands per second. the amount.
Glossary
- Throughput (TPS): Throughput refers to the amount of data successfully transmitted per unit time (measured in bits, bytes, packets, etc.) for a network, device, port, virtual circuit or other facility