RocketMQ (three) technical architecture

1 Technical architecture

RocketMQ architecture is mainly divided into four parts, as shown in the figure above:

• Producer: The role of message publishing, supporting distributed cluster deployment. Producer selects the corresponding Broker cluster queue for message delivery through the load balancing module of MQ. The delivery process supports fast failure and low latency.
• Consumer: The role of message consumption, supporting distributed cluster deployment. It supports two modes of push and pull to consume messages. At the same time, it also supports the consumption of cluster mode and broadcast mode. It provides a real-time message subscription mechanism to meet the needs of most users.
• NameServer: NameServer is a very simple Topic routing registration center. Its role is similar to zookeeper in Dubbo and supports dynamic registration and discovery of Broker. It mainly includes two functions: Broker management, NameServer accepts the registration information of the Broker cluster and saves it as the basic data of routing information. Then provide a heartbeat detection mechanism to check whether the Broker is still alive; routing information management, each NameServer will save the entire routing information about the Broker cluster and queue information for client queries. Then the Producer and Conumser can know the routing information of the entire Broker cluster through NameServer, so as to deliver and consume messages. NameServer is usually deployed in a cluster, and the instances do not communicate with each other. Broker registers its own routing information with each NameServer, so each NameServer instance saves a complete routing information. When a NameServer goes offline for some reason, the Broker can still synchronize its routing information with other NameServers, and Producer and Consumer can still dynamically perceive Broker's routing information.
• BrokerServer: Broker is mainly responsible for message storage, delivery and query, as well as service high availability guarantee. In order to realize these functions, Broker includes the following important sub-modules.

1. Remoting Module: The entity of the entire Broker, responsible for processing requests from clients.
2. Client Manager: Responsible for managing the client (Producer/Consumer) and maintaining the Topic subscription information of the Consumer
3. Store Service: Provides a convenient and simple API interface to process the message storage to the physical hard disk and query functions.
4. HA Service: Highly available service, providing data synchronization function between Master Broker and Slave Broker.
5. Index Service: Index service for messages delivered to Broker according to a specific Message key to provide quick query of messages.

2 deployment architecture

[External link image transfer failed. The origin site may have an anti-hotlink mechanism. It is recommended to save the image and upload it directly (img-6MCviSPf-1606900409585)(https://github.com/apache/rocketmq/raw/master/docs/cn /image/rocketmq_architecture_3.png)]

RocketMQ network deployment characteristics

• NameServer is an almost stateless node that can be deployed in clusters without any information synchronization between nodes.
• Broker deployment is relatively complicated. Broker is divided into Master and Slave. One Master can correspond to multiple Slaves, but one Slave can only correspond to one Master. The correspondence relationship between Master and Slave is defined by specifying the same BrokerName and different BrokerId. BrokerId is 0 Represents Master, non-zero represents Slave. Master can also deploy multiple. Each Broker establishes a long connection with all nodes in the NameServer cluster, and regularly registers Topic information to all NameServers. Note: The current RocketMQ version supports one Master and multiple Slaves in the deployment architecture, but only the slave server with BrokerId=1 will participate in the message reading load.
• The Producer establishes a long connection with one of the nodes (selected at random) in the NameServer cluster, periodically obtains Topic routing information from the NameServer, establishes a long connection to the Master that provides the Topic service, and sends heartbeats to the Master regularly. Producer is completely stateless and can be deployed in clusters.
• The Consumer establishes a long connection with one of the nodes (selected at random) in the NameServer cluster, periodically obtains Topic routing information from the NameServer, establishes a long connection to the Master and Slave that provide Topic services, and sends heartbeats to the Master and Slave regularly. Consumers can subscribe to messages from Master or Slave. When consumers pull messages from Master, the Master server will determine whether to read old messages or not according to the distance between the pull offset and the maximum offset. I/O), as well as whether the slave server is readable or not, it is recommended to pull from Master or Slave next time.

Combined with the deployment architecture diagram, describe the cluster workflow:

• Start the NameServer, listen to the port after the NameServer gets up, and wait for the Broker, Producer, and Consumer to connect, which is equivalent to a routing control center.
• Broker starts, maintains long connections with all NameServers, and sends heartbeat packets regularly. The heartbeat packet contains the current Broker information (IP + port, etc.) and stores all Topic information. After successful registration, there is a mapping relationship between Topic and Broker in the NameServer cluster.
• Before sending and receiving messages, create a topic. When creating a topic, you need to specify which broker the topic will be stored on, or you can automatically create a topic when sending a message.
• Producer sends a message, and establishes a long connection with one of the NameServer clusters when it starts, and obtains from the NameServer which Brokers the currently sent Topic exists on, polls and selects a queue from the queue list, and then establishes with the broker where the queue is located The long connection sends a message to the Broker.
• Consumer is similar to Producer. It establishes a long connection with one of the NameServers, obtains which Brokers currently subscribed to, and then directly establishes a connection channel with the Broker to start consuming messages.

3 Original link

注释：来源于GitHub
https://github.com/apache/rocketmq/blob/master/docs/cn/README.md