[RocketMQ advanced one] RocketMQ principle and architecture

RocketMQ core component diagram

RocketMQ is an open source message middleware, which is mainly composed of NameServer, Producer, Broker, and Consumer.

NameServer : NameServer is mainly responsible for the management of Topic and routing information, and its function is similar to Dubbo's zookeeper.
Producer : The message producer, responsible for generating messages, generally the business system is responsible for generating messages.
Broker : Message relay role, responsible for storing and forwarding messages.
Consumer : Message consumer, responsible for message consumption, generally the background system is responsible for asynchronous consumption.

RokcetMQ physical deployment diagram

NameServer : NameServer is an almost stateless node that can be deployed in clusters without any information synchronization between nodes.

Broker : Divided into Master and Slave. One Master can correspond to multiple Slaves, but one Slave can only correspond to one Master. The correspondence between Master and Slave is defined by specifying the same BrokerName and different BrokerId. A BrokerId of 0 means Master. 0 means Slave. Master can also deploy multiple. Each Broker establishes a long connection with all nodes in the Name Server cluster, and regularly registers Topic information to all Name Servers. Master supports reading and writing, and Slave only supports reading .

Producer : The Producer establishes a long connection with one of the nodes (selected at random) in the Name Server cluster, periodically fetches Topic routing information from the Name Server, establishes a long connection to the Master that provides the Topic service, and sends heartbeats to the Master regularly . Producer is completely stateless and can be deployed in clusters .

Consumer : Consumer establishes a long connection with one of the nodes in the Name Server cluster (selected at random), regularly fetches Topic routing information from the Name Server, and establishes a long connection to the Master and Slave that provide Topic services , and sends heartbeats to the Master and Slave regularly . Consumer can subscribe to messages from Master or Slave, and the subscription rules are determined by Broker configuration.

RocketMQ logical deployment structure

Producer Group

Used to represent a message sending application, a Producer Group contains multiple Producer instances, which can be multiple machines, multiple processes of a machine, or multiple Producer objects of a process. A Producer Group can send multiple Topic messages. The functions of the Producer Group are as follows:

Identify a type of Producer
You can query multiple Producer instances under this messaging application through operation and maintenance tools
When sending distributed transaction messages, if the Producer goes down unexpectedly midway, Broker will actively call back any machine in the Producer Group to confirm the transaction status.

Consumer Group

Used to represent a consumer messaging application, a Consumer Group contains multiple Consumer instances, which can be multiple machines, multiple processes, or multiple Consumer objects of a process. Multiple Consumers under a Consumer Group consume messages in an amortized manner. If it is set to broadcast, then each instance under this Consumer Group consumes the full amount of data.

NameServer routing registration and deletion mechanism

Broker sends a heartbeat packet to NameServer every 30 seconds, and the heartbeat packet contains routing information for the topic
NarneServer updates the information in brokerLiveTable after receiving the Broker heartbeat packet, especially recording the heartbeat time lastUpdateTime
NarneServer scans the brokerLiveTable every 10s, detects the last time the heartbeat packet was received in the table, compares the current time with the last time, if it exceeds 120s , the broker is considered unavailable, and all information related to the broker in the routing table is removed
The message producer pulls the topic routing information, that is, the message producer does not immediately perceive the addition and deletion of the Broker server .

Note : Producer (message producer), first of all, it has to know which Broker the message is to be sent to, so every 30s it will obtain the Topic and Broker mapping relationship from a NameServer and store it in local memory . If a new Broker is found, it will be established with it Long connection, heartbeat will be sent to Broker every 30s to maintain the connection . And it will poll the currently available Broker to send messages to achieve the purpose of load balancing :

Synchronous sending , if the sending fails, it will retry twice by default (retryTimesWhenSendFailed = 2) , and the broker that failed last time will not be selected , and the delivery will be sent to other brokers .
Asynchronous sending , it will retry if it fails. The default is twice (retryTimesWhenSendAsyncFailed = 2), but only on the same Broker .

RocketMQ's message domain model diagram

Topic

Topic represents the first-level type of messages. For example, the messages of an e-commerce system can be divided into: transaction messages, logistics messages, etc. A message must have a topic.
The most fine-grained subscription unit, a group can subscribe to multiple topic messages.

Tag : indicates the second-level type of messages. For example, transaction messages can be divided into: transaction creation messages, transaction completion messages, etc. RocketMQ provides two-level message classification for convenient and flexible control.

Group group : A group can subscribe to multiple topics.

Message Queue : The physical management unit of the message. There can be multiple Queues under a Topic, and the introduction of Queues enables the storage of messages to be distributed and clustered, with horizontal scalability.

In RocketMQ, all message queues are persistent data structures with unlimited length. The so-called unlimited length means that each storage unit in the queue is of fixed length. The storage unit is accessed using Offset, and the offset is java long type. , 64-bit, theoretically will not overflow within 100 years, so it is considered to be unlimited in length.

You can also think of Message Queue as an array of unlimited length, and Offset is the subscript.

Sequence message schematic

The order of consuming messages must be the same as the order of sending messages. In RocketMQ, the main part is the local order, that is, a type of message must be sent in sequence in a single thread by the Producer and sent to the same queue, so that the Consumer can follow Producer consumes messages in the order in which they are sent.

RocketMQ message storage design principle diagram

CommitLog : Message storage file, all messages of message subject are stored in CommitLog file. The logical view of Commitlog file storage is shown in the figure

ConsumeQueue

Message consumption queue. After the message reaches the CommitLog file, it will be asynchronously forwarded to the message consumption queue for consumption by message consumers. The storage format of ConsumeQueue is as follows:

A single ConsumeQueue file contains 300,000 entries by default. The length of a single file is 30w × 20 bytes. A single ConsumeQueue file can be seen as an array of ConsumeQueue entries. Its subscript is the logical offset of the ConsumeQueue, and the message consumption progress is stored The offset is the logical offset.
ConsumeQueue is the index file of the Commitlog file, and its construction mechanism is that when the message arrives in the Commitlog file, a special thread generates a message forwarding task to build the message consumption queue file and the index file mentioned below.

IndexFile : message index file, which mainly stores the correspondence between message Key and Offset.

The message consumption queue is an index file specially built by RocketMQ for message subscriptions, which improves the speed of retrieving messages based on topics and message queues. In addition, RocketMQ introduces a Hash index mechanism to index messages. The design of HashMap includes two basic points: Hash slot and Hash Conflicting linked list structure. RocketMQ index file layout is shown in the figure

lndexFile contains lndexHeader, Hash slot, and Hash entries in total

Transaction status service

Store the transaction status of each message.

Timed message service

Each delay level corresponds to a message consumption queue, which stores the message pull progress of the delay queue.

RMQ file storage model layer

RocketMQ business processor layer

The Broker side reads and writes the business logic entry of the message. This layer mainly contains business logic related processing operations (based on the analysis of the RequestCode in the RemotingCommand to distinguish specific business operation types, and then execute different business processing procedures), Such as pre-checking and verification steps, constructing MessageExtBrokerInner object, decoding deserialization, constructing Response return object, etc.

RocketMQ data storage component layer

This layer is mainly RocketMQ's storage core class—DefaultMessageStore, which is the access entry for RocketMQ message data files. The "putMessage()" and "getMessage()" methods of this class are used to read and write the log data files stored by CommitLog messages. Operation (the specific read-write access operation still depends on the method provided by the CommitLog object model in the next layer);
In addition, when the component is initialized, many storage-related background service threads will be started, including AllocateMappedFileService (MappedFile pre-allocation service thread), ReputMessageService (playback storage message service thread), HAService (Broker master-slave synchronization high availability service thread), StoreStatsService (message storage statistics service thread), IndexService (index file service thread), etc.

RocketMQ storage logical object layer

This layer mainly contains three model classes IndexFile, ConsumerQueue and CommitLog directly related to RocketMQ data file storage.
IndexFile provides access services for index data files, ConsumerQueue provides access services for logical message queues, and CommitLog provides access services for log data files stored in messages.
These three model classes also constitute the overall structure of the RocketMQ storage layer.

Encapsulated file memory mapping layer

RocketMQ mainly uses MappedByteBuffer and FileChannel in JDK NIO to complete the reading and writing of data files.
Among them, MappedByteBuffer is used to complete the reading and writing of large files in a memory-mapped disk file method, and this class is encapsulated into a MappedFile class in RocketMQ.
Here, each type of single file is provided by the MappedFile class to provide read and write operation services (among which, the MappedFile class provides sequential write/random read, memory data flashing, memory cleaning, etc. and file-related services).

Disk storage layer

Mainly refers to the disk used to deploy RocketMQ server. Here, you need to consider the impact of different disk types (such as SSD or ordinary HDD) characteristics and disk performance parameters (such as IOPS, throughput, and access latency) on sequential write/random read operations.

Message refresh in RocketMQ

Message brushing in RocketMQ can be divided into two types: synchronous brushing and asynchronous brushing.

Synchronous flashing

When the write success status is returned, the message has been written to the disk.
The specific process is that after the message is written into the PAGECACHE of the memory, it immediately informs the flashing thread to flash, and then waits for the flashing to complete. After the flashing thread is executed, the waiting thread is awakened, and the message is successfully written back.
Generally only used in financial scenarios.

Asynchronous brushing

When the write success status is returned, the message may only be written to the PAGECACHE of the memory. The write operation returns quickly and the throughput is large; when the amount of messages in the memory accumulates to a certain extent, the disk write operation is triggered uniformly and written quickly.

Original: https://blog.csdn.net/zhuyanlin09/article/details/101751549