The message data storage in RocketMQ uses zero-copy technology (using mmap + write), and the file system uses the Linux Ext4 file system for storage.
Storage of message data
In RocketMQ, message data is stored in disk files. In order to ensure the performance of writing, RocketMQ guarantees sequential writing as much as possible. The efficiency of sequential writing is much higher than that of random writing.
The storage of RocketMQ messages is completed by the cooperation of ConsumeQueue and CommitLog. CommitLog is the file that actually stores the data, and ConsumeQueue is the index file, and the stored data points to the configuration of the physical file.
As shown in FIG:
- The message body and metadata are stored in CommitLog
- Consume Queue is equivalent to the partition in Kafka. It is a logical queue that stores the initial offset, log size and MessageTag hashCode of this Queue in CommiLog.
- Each time the message queue is read, the consumerQueue is read first, and then the message body is obtained from the commitLog through the consumerQueue.
The location of the file
Synchronous flashing and asynchronous flashing
RocketMQ, in order to improve performance, will try to ensure the sequential write of the disks as much as possible. When a message is written to RocketMQ through the Producer, there are two ways to write to the disk, namely, synchronous flashing and asynchronous flashing.
Synchronous flashing
- When the write success status is returned, the message has been written to the disk.
- The specific process is: after the message is written into the PAGECACHE of the memory, it immediately informs the flashing thread to flash, and then waits for the flashing to complete. After the flashing thread is executed, the waiting thread is awakened, and the message is written successfully.
Asynchronous brushing
- When the write success status is returned, the message may just be the PAGECACHE written into the memory. The write operation returns quickly and the throughput is large.
- When the amount of messages in the memory accumulates to a certain extent, the disk writing action is triggered uniformly and quickly written.
Specify the flashing method in the broker configuration file
- flushDiskType=ASYNC_FLUSH -- 异步
- flushDiskType=SYNC_FLUSH -- 同步
