[RocketMQ] Design Ideas, Architecture Model

 

1. Design of message middleware

• You can start thinking from the basic needs
  • The basic thing is to be able to support the sending and receiving of messages. If network communication is involved, NIO must be involved.
  • Message storage in the message center (persistent/non-persistent)
  • Serialization and deserialization of messages
  • Cross-language
  • Message confirmation mechanism, how to avoid message retransmission
• Advanced Features
• Orderliness of messages
• Whether to support transaction messages
• Messaging performance, support for high concurrency and large data volume
• Whether to support cluster
• Reliable storage of messages
• Whether to support multiple protocols

2. MQ message storage options

Judging from the storage methods used by several mainstream MQ message queues, there are mainly three

1. Distributed KV storage, such as levelDB and Redis used in ActiveMQ. This storage method can be used when the message read and write ability is not high.
2. File system storage, such as kafka, RocketMQ, RabbitMQ, use message flashing to the file system on the deployed machine for persistence. This solution is suitable for message middleware with high throughput requirements, because message flashing Disk is a high-efficiency, high-reliability, and high-performance persistence method. Unless the disk fails, there will generally be no problems that cannot be persisted.
3. Relational databases, such as ActiveMQ, can use mysql as message storage. Relational databases will have bottlenecks in IO performance when the amount of data in a single table reaches tens of millions. Therefore, ActiveMQ is not suitable for high-throughput message queue scenarios.

In general, for storage efficiency, file systems are better than distributed KV storage, and distributed KV storage is better than relational databases

3. Introduction to RocketMQ

RocketMq is a message middleware open sourced by Alibaba. It was open sourced in 2012 and became the top Apache project in 2017.

Its core design draws on Kafka, so when we understand RocketMQ, we will find many of the same features as Kafka. At the same time, Rocket is quite different from Kafka in some functions. The following are the characteristics of RocketMQ:

1. Support cluster model, load balancing, horizontal scalability
2. Billion-level message accumulation capability
3. Using the principle of zero copy, write disk sequentially and read randomly
4. The underlying communication framework uses Netty NIO
5. NameServer replaces Zookeeper to implement service addressing and service coordination
6. Message failure retry mechanism, message can be queried
7. Emphasize that the cluster has no single point, is scalable, is highly available at any point, and is horizontally scalable
8. After many double eleven tests

4.RocketMQ architecture

There is nothing special about the cluster itself, which is similar to the overall architecture of kafka, where zookeeper is replaced with NameServer.

In the early version of rocketmq (2.x), there was no namesrv component, and zookeeper was used for distributed coordination and service discovery. However, in the later period, Alibaba Data made improvements and optimizations based on actual business needs, and developed a self-organized lightweight Class-level namesrv is used to register the client service and Broker's request routing work. Namesrv does not store any message location. Frequent operation of zookeeper location storage data will affect the overall cluster performance

RocketMQ consists of four parts:

• Name Server: Provides lightweight service discovery and routing, and can be deployed in clusters without any information synchronization between nodes.

• Broker: The role of message transfer, responsible for storing and forwarding messages, and the deployment is relatively complicated. Broker is divided into Master and Slave

  • The correspondence between Master and Slave is defined by specifying the same BrokerName but different BrokerId. A BrokerId of 0 means Master, and non-zero means Slave.
  • One Master can correspond to multiple Slaves, but one Slave can only correspond to one Master. Multiple masters can also be deployed.

• Producer: Producer

  • Producers with the same Producer Group form a cluster,
  • Establish a long connection with one of the nodes in the Name Server cluster (selected at random), and regularly fetch Topic routing information from the Name Server
  • Establish a long connection to the Master that provides Topic services, and send heartbeats to the Master regularly.
  • Producer is completely stateless and can be deployed in clusters.

• Consumer: Consumer, an instance of receiving messages for consumption

  • Consumers with the same Consumer Group form a cluster
  • Establish a long connection with one of the nodes in the Name Server cluster (selected at random), and regularly fetch Topic routing information from the Name Server
  • Establish long connections to Master and Slave that provide Topic services, and send heartbeats to Master and Slave regularly.

  Consumer can subscribe to messages from Master or Slave, and the subscription rules are determined by Broker configuration.

To use rocketmq, at least two processes need to be started, nameserver and broker. The former is the registration center for various topics, and the latter is the real broker.