Routing Center (NameServer)
NameServers are deployed through clusters, in order to pursue simplicity and efficiency, NameServers do not communicate with each other.
NameServer maintains a long connection with each Broker server, and checks whether the Broker is alive at intervals of 30s. If it detects that the Broker is down, it will be removed from the routing registry. However, routing changes will not immediately notify the message producer, reducing the complexity of the NameServer implementation, and providing a fault-tolerant mechanism at the message sending end to ensure high availability of message sending.
Message producer (Producer)
Message producer cluster deployment.
The topic routing information is cached locally. If the topic routing information is not cached in the local routing table, a request to obtain routing information is sent to the NameServer, the local routing information table is updated, and the message producer updates the routing table from the NameServer every 30s.
Message sending selects Broker through load balancing strategy to avoid excessive pressure on a single server.
Message sending exception mechanism, message sending high availability mainly through two means: retry and Broker evasion. Broker evasion is to find an error during a message sending process. In a certain period of time, the message producer will not select the message queue on the Broker to improve the success rate of sending messages.
Batch message sending supports sending multiple messages under the same topic to the message server at once.
Consumer
Message consumer cluster deployment.
Load balance when consumers consume messages. By default, the RebalanceService thread performs message queue load every 20s. According to the number of consumers in the current consumer group and the number of topic queues, the queue is allocated according to a certain load algorithm. The allocation principle is that the same consumer can allocate multiple message consumption queues. , The same message consumption queue can only be allocated to one consumer at the same time.
The message pull is pulled by the PullMessageService thread according to the pull task created by the RebalanceService thread. By default, 32 messages are pulled in a batch and submitted to the consumer consumption thread pool to continue the next message pull. If message consumption is too slow, message accumulation will trigger message consumption pull flow control.
Server (Broker)
Zero copy technology.
The server-side high-concurrency reading and writing mainly uses the PageCache feature of the Linux operating system, and directly operates the PageCache through Java's MappedByteBuffer. MappedByteBuffer can directly map the file to the memory, and hit PageCahe as much as possible through sequential writing to the disk (Commitlog), skip reading, and pre-reading data, thereby greatly reducing disk IO.
Commitlog, message storage file, RocketMQ uses a single file to store messages of all topics in order to ensure the high throughput of message sending, ensuring that the message storage is written in complete order, but this also brings inconvenience to file reading. For this reason, RocketMQ To facilitate message consumption, a message consumption queue file (ConsumerQueue) is constructed, which is organized based on topics and queues. At the same time, RocketMQ implements a Hash index file (IndexFile) for messages, which can set index keys for messages, and quickly retrieve messages from Commitog files based on the index . When the message arrives in the Commitlog file, the message will be forwarded to the message consumption queue file and index file in near real-time through the ReputMessageService thread. For the sake of safety, RocketMQ introduces the abort file to record whether the Broker is shut down normally or abnormally. When restarting the Broker, in order to ensure the correctness of the Commitlog file, the message consumption queue file and the Hash index file, different strategies are adopted to restore the file. .