Preface
Because of the high reliability requirements of distributed queues, data must be stored persistently.
- The message producer sends the message
- MQ receives the message, persists the message, and adds a new record in the storage
- Return ACK to the producer
- MQ push message to the corresponding consumer, and then wait for the consumer to return ACK
- If the message consumer returns ack successfully within the specified time, then MQ considers that the message consumption is successful and deletes the message in the storage, that is, step 6; if MQ does not receive an ACK within the specified time, it considers that the message consumption has failed and will try Push the message again and repeat steps 4, 5, and 6
- MQ delete message
Storage medium
- Relational database DB
Another open source MQ under Apache—ActiveMQ (KahaDB used by default for message storage) can choose JDBC to do message persistence, and JDBC message storage can be realized through simple xml configuration information. Because ordinary relational databases (such as Mysql) often have bottlenecks in their IO read and write performance when the amount of data in a single table reaches tens of millions. In terms of reliability, this solution relies heavily on DB. Once DB fails, MQ messages cannot be stored on disk, which will cause online failures.
- File system
At present, several commonly used products in the industry (RocketMQ/Kafka/RabbitMQ) all use message flashing to the file system of the deployed virtual machine/physical machine for persistence (flashing can generally be divided into asynchronous flashing and synchronous flashing. Disk in two modes). Message flash disk provides a high-efficiency, high-reliability and high-performance data persistence method for message storage. Unless the MQ machine itself is deployed or the local disk is hung up, failures that cannot be persisted will generally not occur.
Performance Comparison File System>Relational Database DB
Storage and sending of messages
Message storage
If the disk is used properly, the speed of the disk can completely match the data transmission speed of the network. The current high-performance disks can achieve a sequential write speed of 600MB/s, which exceeds the transmission speed of a general network card. But the random write speed of the disk is only about 100KB/s, which is 6000 times different from the sequential write performance! Because of such a huge speed difference, a good message queuing system will be many orders of magnitude faster than a normal message queuing system. RocketMQ's messages are written in sequence, which ensures the speed of message storage.
Message sending
Linux operating system is divided into [user mode] and [kernel mode]. File operations and network operations need to involve switching between these two modes, and data copying is inevitable.
A server sends the contents of the local disk file to the client, generally divided into two steps:
1) read; read the contents of a local file;
2) write; send the read content through the network.
These two seemingly simple operations actually performed 4 data copies, namely:
- Copy data from disk to kernel mode memory;
- Copy from kernel mode memory to user mode memory;
- Then copy from the user mode memory to the kernel mode memory of the network driver;
- Finally, the kernel mode memory of the network driver is copied to the network card for transmission.
By using mmap, the memory copy to the user mode can be omitted, and the speed can be improved. This mechanism is implemented in Java through MappedByteBuffer
RocketMQ makes full use of the above characteristics, which is the so-called "zero copy" technology, to improve the speed of message storage and network transmission.
It should be noted here that there are several limitations to the memory mapping method using MappedByteBuffer, one of which is that only 1.5~2G files can be mapped to user-mode virtual memory at a time. This is why RocketMQ sets a single CommitLog log data file by default For the 1G reason
Message storage structure
RocketMQ message storage is completed by the cooperation of ConsumeQueue and CommitLog. The real physical storage file of the message is CommitLog. ConsumeQueue is the logical queue of messages, similar to the index file of the database, and the storage is the address pointing to the physical storage. Each Message Queue under each topic has a corresponding ConsumeQueue file
As mentioned above, the CommitLog log data file is 1G. It is set so large here because this storage file includes the metadata information of Topic, QueueId, and Message. After 1G is full, a 1G file will be automatically created for storage! Then it will be troublesome to retrieve data from a 1G file here, and the performance is definitely not good. Then it can reflect the role of ConsumeQueue. The index of the message is stored in ConsumeQueue. For example, when consumers need to consume a certain message, it is not directly. To query in the file of CommitLog1G, first get the index of the message in ConsumeQueue, find the message position corresponding to the index, and then go to CommitLog to accurately locate a certain piece of data. This processing method can speed up the reading of CommitLog Speed, the message content is not stored in ConsumeQueue, only the index is stored, ConsumeQueue is a MessageQueue one-to-one correspondence, ConsumeQueue is in memory, if the system is down, it will be retrieved from CommitLog when restarting.
- CommitLog: Store the metadata of the message
- ConsumerQueue: Index of stored messages in CommitLog
- IndexFile: For message query, a method of querying messages by key or time interval is provided. This method of finding messages through IndexFile does not affect the main process of sending and consuming messages. When we consume messages, we mainly use ConsumerQueue!
The CommitLog here is actually the message storage directory configured in our broker configuration file, as shown in the figure below. Here you can also see that the file size is 1G! There is not only one CommitLog file, when the stored data is greater than 1G, a file will be created for storage!
The figure below is ConsumerQueue , each Message Queue under each topic has a corresponding ConsumeQueue file
IndexFile as shown below
Swipe mechanism
RocketMQ's messages are stored on the disk, which not only guarantees the recovery after power failure, but also allows the amount of stored messages to exceed the memory limit. In order to improve performance, RocketMQ will try its best to ensure the sequential write of the disk. When messages are written to RocketMQ through the Producer, there are two ways to write to disk, distributed synchronous flashing and asynchronous flashing.
Synchronous flash disk
When the write success status is returned, the message has been written to the disk. The specific process is that after the message is written into the PAGECACHE of the memory, it immediately informs the flashing thread to flash, and then waits for the flashing to complete. After the flashing thread is executed, the waiting thread is awakened and the message is written successfully.
When the asynchronous flash disk
returns the write success status, the message may only be the PAGECACHE written into the memory. The write operation returns quickly and the throughput is large; when the amount of messages in the memory accumulates to a certain extent, the disk write action is triggered uniformly, which is fast Write.
So which mode is used in the actual enterprise, we must consider the trade-off between security (synchronous) and performance (asynchronous).
Synchronous flashing or asynchronous flashing is set through the flushDiskType parameter in the Broker configuration file. This parameter is configured as one of SYNC_FLUSH and ASYNC_FLUSH.
