Kafka learns a very detailed classic tutorial.
So unlike the traditional design of caching data in memory and then flashing to the hard disk, Kafka directly writes the data to the log of the file system.
Constant time operation efficiency
In most messaging systems, the mechanism of data persistence is often to provide a B-tree or other random read-write data structure for each cosumer. B-tree is of course great, but it also comes with some costs: for example, the complexity of B-tree is O(log N), O(log N) is usually considered to be constant complexity, but this is not the case for hard disk operations. . It takes 10ms for a disk to perform a search, and each hard disk can only be searched once at the same time, so concurrent processing becomes a problem. Although the storage system uses the cache to make a lot of optimizations, the observations on the performance of the tree structure show that its performance tends to decrease linearly with the growth of data. When the data doubles, the speed will be doubled.
Intuitively speaking, for the message system mainly used for log processing, the persistence of data can be realized simply by appending the data to the file, and just read it from the file when reading. The advantage of this is that both read and write are O(1), and read operations will not block write operations and other operations. The performance advantage brought by this is very obvious, because the performance has nothing to do with the size of the data.
Since it is possible to use hard disk space with almost no capacity limitation (as opposed to memory) to establish a message system, it can provide some features that are not available in general message systems without performance loss. For example, the general message system deletes the message immediately after it is consumed, but Kafka can save the message for a period of time (such as one week), which provides consumers with good mobility and flexibility. This will be discussed in future articles. There will be details.