RocketMQ is a message middleware commonly used by Java brothers. Although it is commonly used, RocketMQ architecture is often forgotten. There are two reasons for this: being busy with business development and then forgetting it if you don’t read it for a long time, and not understanding the root cause of the architecture design and not being able to remember it well. This article describes the architecture design process in vernacular, keeping in mind the RocketMQ architecture.
1. The thinking process of architecture
First of all, when remembering the principles and architecture of the framework, you must first grasp the overall context, think about why it is designed this way, and finally think about the details, so that you can remember it firmly. This article explains step by step the reasons for the design of RocketMQ architecture by asking questions step by step.
1.Basic form
(1) If you were a RocketMQ developer and asked to design a message middleware, what roles would you design?
Answer: At least 3 characters must be designed:
- Message transfer station: Broker, Broker is the core and is responsible for: receiving messages, storing messages, processing consumer consumption requests, backup and disaster recovery, etc.
- Producer: Producer, produces messages and then delivers them to Broker.
- Consumer: Consumer, consumes messages from Broker.
2. How to save messages
(2) After having the basic form, we know that the specific message must be stored in the Broker, so how should the message be stored in the Broker?
Answer: This is based on real-life cases. For example, when a logistics company sends express delivery, the express delivery to the same city must be arranged together, and then transported to that city with the same batch of trucks. In this way, the entire logistics system operates most efficiently. Clustering is used here to bring similar things together.
Similarly, when designing how to store messages, we also use the concept of clustering. We put messages of the same type into a logical space, and this logical space is the topic.
(3) What is the internal structure of Topic?
Answer: There must be message objects inside Topic. So in what data structure do these message objects exist together? The message sent first must be consumed first. The first-in-first-out data structure-queue is used here. This is the message queue MessageQueue. Therefore, Topic is internally composed of MessageQueue, and message objects are stored inside the message queue.
3.Introduce clusters
(4) We know that Broker is the core of RocketMQ. What should we do if such an important core fails?
Answer: Since it is the core of RocketMQ, it must ensure high availability and cannot be down, so RocketMQ will deploy multiple Brokers to form a cluster to provide external services.
4. Let’s talk about how to save the message
(5) In order to ensure high availability, RocketMQ will deploy multiple Brokers to form a cluster. If there are multiple machines in a cluster scenario, how to save Topics?
Answer: We must learn the idea of "don't put eggs in one basket". Since a large amount of messages need to be stored and there are multiple Brokers, in order to share the performance pressure of a single machine, share the pressure of storage capacity, and ensure data disaster recovery, different topics are stored in different Brokers.
Still following the logistics example above, for example, express delivery from Beijing to Nanjing must be transported by the same batch of trucks. If the express delivery is small, one truck will be used, and if the express delivery is large, multiple trucks will be used. The express delivery is divided into multiple trucks. Similarly, Topics in RocketMQ are also scattered and stored on multiple Brokers, and the message content stored on each Broker is different.
(6) If different topics are stored in different brokers, the data of a certain topic may be too large, and the data skew will directly destroy a certain broker. What should I do?
Answer: As we mentioned above, Topic is actually a collection of queues. Then you only need to store the queues in different Brokers.
(7) If different topics are stored in different Brokers, there is still a risk of data loss, but the data lost for a certain topic will become smaller. How to do data disaster recovery backup in this case?
Answer: At this time, the Broker's master-slave architecture will be used. The Broker is divided into Master and Slave according to their roles. Data synchronization is performed between the master and slave regularly. The Master is responsible for responding to the client's read and write requests, storing messages, processing consumer requests, etc., while the Slave is only responsible for synchronizing the Master's data.
5. Talk about NameServer
(8) Since Brokers are clusters, when delivering messages, producers must know which Brokers there are and which Broker to deliver messages to. How to do this?
Answer: RocketMQ introduces the concept of NameServer. NameServer is equivalent to the big housekeeper. It knows all the basic information in RocketMQ. NameServer stores the metadata of the RocketMQ cluster. The metadata stored in NameServer mainly includes:
- What Brokers are there in the cluster?
- Who are the producers?
- Who are the consumers?
- What topics are there in the cluster?
- On which Brokers do these Topic message queues exist?
(9) How does the Nameserver know these messages?
Answer: It’s similar to when someone went to work as an errand in the government in ancient times. Before going on an errand, he had to register all his information in a register. Similarly, Broker, Producer, and Consumer will also register data to NameServer when starting.
The Broker will register itself with the NameServer when it starts and continuously update metadata through heartbeats. Similarly, Producer and Consumer will also establish connections with NameServer and dynamically interact with data in the cluster, making it easy to report their own information and obtain other information in the cluster.
At this point, RocketMQ's architecture diagram has taken shape, and the reason for each component's design is also clear.
2. Summary
There are four core roles in RocketMQ: Broker, Producer, Consumer, and NameServer. There are two core objects for message storage: Topic and MessageQueue.
In order to ensure that data is not lost and data is not skewed, MessageQueue in the same Topic will be stored in different Brokers.