Share the internal top-secret RocketMQ core principles and best practice notes compiled by Ali experts!

The source code of this article is based on RocketMQ 4.2.0 and RocketMQ 4.3.0, from the actual use of RocketMQ to RocketMQ source code analysis, and then to RocketMQ enterprise landing practice plan, step by step explanation. Make readers understand RocketMQ from the shallower to the deeper.

In the process of source code analysis, this article first talks about the overall process, and then explains in detail according to the modules and steps. I hope readers can draw inferences when reading, and know what is and why.

There are nine chapters in this article, divided into five parts. The first part explains the introduction of message queues and the principles and best practices of RocketMQ production and consumption; the second part explains the RocketMQ architecture from an overall perspective; the third part explains the basic principles of each component of RocketMQ; The part goes deep into RocketMQ, explaining how to read the source code and how to carry out enterprise practice; the fifth part is an appendix, including Namesrv, Broker's core parameter configuration instructions and Exporter monitoring indicator comments.

 

table of Contents

 

main content

Overview of RoketMQ;

• 1.1 What is a message queue, Message Queue, in a broad sense, is a message queue service middleware that provides a complete set of software systems for information production, delivery, and consumption.
• 1.2 Why do we need a message queue? Through the explanation in the previous section, I believe that readers have a preliminary understanding of message queues, so when do we usually use message queues?
• 1.3 Common message queues,
• 1.4 The history and future of RocketMQ, Apache RocketMQ is an open source, distributed message delivery and streaming data platform. Born from Alibaba, after experiencing 3 versions within Alibaba, it has been one of the top open source projects of Apache until now. There are 10000+star, 5000+fork, 170+contributors (developers who submit code on GitHub and are adopted) on GitHub. The current latest version is version 4.7.0 in March 2020. This section mainly introduces the generation and growth process of RocketMQ.

 

The producer principles and best practices of RocketMQ;

• 2.1 The principle of producer. Through the explanation in Chapter 1, I believe that readers have a basic understanding of RocketMQ. This section will give a basic introduction to the producers in RocketMQ.
• 2.2 Producer startup process, DefaultMQProducer is the default producer implementation in RocketMQ, the inheritance relationship between the classes of DefaultMQProducer, you can see that this producer includes the producer's operation and configuration properties when it is implemented. This is a typical class object design . This section will introduce some core properties and methods of class objects.
• 2.3 Message sending process, the message sending process first is that the RocketMQ client receives the business layer message, and then sends an RPC request to the Broker through DefaultMQProducerImpl, and then the Broker processes the request and saves the message.
• 2.4 Best practices for sending messages, sending ordinary messages, sequential messages, delayed messages, transaction messages, one-way messages, and batch messages.
• 2.5 Summary of best practices for producers. Compared with consumers, the use of producers is simpler. The general readers mainly focus on message types, message sending methods and sending parameters, and they can use RocketMQ to send messages normally.
  How to choose the message type and consumption sending method in actual use? The author here summarizes the commonly used message types, message sending methods, and basic sending parameters for your reference.

 

RocketMQ's consumption process and best practices;

• 3.1 Consumer overview. Consumers generally refer to a series of code implementations that obtain messages and forward them to business code for processing. How does the consumer behavior work in RocketMQ? This section will describe in detail.
• 3.2 Consumer startup mechanism, RocketMQ client has two independent consumer implementation classes: org.apache.rocketmq.client.consumer.DefaultMQPullConsumer and org.apache.rocketmq.client.consumer.DefaultMQPushConsumer. They will be introduced separately below.
• 3.3 Consumer's Rebalance mechanism, the client is highly reliable through the Rebalance service. When various emergencies such as Broker drop, consumer instance drop, topic expansion, etc. occur, how do the consumer instances in the consumer group rebalance to support the normal consumption of all queues?
• 3.4 Consumption progress saving mechanism, the location manager will be activated at the same time when the consumer starts, so how exactly is the location managed? RocketMQ has designed two site management methods, remote site management and local site management. During cluster consumption, the site is submitted by the client to Broker for storage, and the specific implementation code is in the RemoteBrokerOffsetStore.java file; during broadcast consumption, the site is saved on the consumer's local disk, and the implementation code is in the LocalFileOffsetStore.java file.
• 3.5 Consumption mode, RocketMQ's consumption mode includes Pull and Push. Pull mode: The user takes the initiative to pull messages, manages the location independently, and can flexibly control the consumption progress and consumption speed, which is suitable for special consumption scenarios such as stream computing and consumption is particularly time-consuming.
  The shortcomings are also obvious, the need to accurately control consumption from the code level, there are certain requirements for developers. In RocketMQ, org.apache.rocketmq.client.consumer.DefaultMQPullConsumer is the default Pull consumer implementation class. Push method: Code access is very simple, suitable for most business scenarios. The disadvantage is poor flexibility. After understanding its consumption principle, it is simple and quick to troubleshoot consumption problems. In RocketMQ, org.apache.rocketmq.client.consumer.DefaultMQPushConsumer is the default Push consumer implementation class.
• 3.6 Message filtering
• 3.7 Summary of consumer best practices. This chapter mainly introduces the consumption process and consumer principles, as well as the design and implementation of filters. There are two points to pay special attention to in the use of Pull and Push: how to troubleshoot when the subscription relationship is inconsistent and cannot be consumed. Here is the experience of the author in the practice process.

 

RocketMQ architecture and deployment best practices;

4.1 RocketMQ architecture, RocketMQ is not only a technology, it is also a system.

4.2 Common deployment topologies and deployment practices. There are 5 commonly used RocketMQ deployment topologies. Different deployment methods have different reliability. When deploying in the company, you can choose according to the needs of the enterprise business, or there are new deployment methods You can also share it with the author and the RocketMQ community.

 

Namesrv；

• 5.1 Overview of Namesrv, Namesrv is mainly used to save metadata and improve the usability of Broker in the RocketMQ system.
  In RPC communication, we usually refer to the service provider as the server, and the end that uses the service as the client. If the server is expanding or shrinking, how does the client perceive it? The common practice in the industry is service registration and discovery. Through registration, you can add more server-side instances that provide services. Of course, some instances are down, and you can also remove them to ensure the reliability of the service. As a provider of RocketMQ services, Broker works on the same principle.
• 5.2 Namesrv architecture, Namesrv components, Namesrv start process, Namesrv stop process;
• 5.3 The routing principle of RocketMQ. When a producer sends a message and a consumer consumes a message, it needs to pull Topic routing information from Namesrv. Then how is this routing information registered to Namesrv? If the Broker goes down abnormally, how is the routing information updated?
  Below, we will explain in detail through two aspects of route registration and route elimination.

 

Broker storage mechanism;

• 6.1 Broker overview, this chapter mainly explains the basic knowledge of Broker and the position of Broker in the system. Lead the reader to look at the Broker's file directory structure from the deployment results, and lay the foundation for the next chapter to learn the storage module. The core content of this chapter is: ● Broker's position in the RocketMQ system. ● Broker's data directory structure. ● Broker's start and stop process.
• 6.2 Broker storage mechanism, accumulation capacity is an important assessment indicator of the message queue. The storage mechanism is the core of RocketMQ, and it is also the highlight design, because the storage mechanism determines the efficiency of writing and querying.
• 6.3 Broker CommitLog index mechanism, most storage components have an index mechanism, RocketMQ also has a huge amount of accumulation capacity, and can speed up reading and query through index.
  This section mainly explains the basic principles of RocketMQ's ConsumeQueue and IndexFile indexes: ● Index data structure. ● The index construction process. ● How to use the index.
• 6.4 Brokeri expired file deletion mechanism, RocketMQ mainly saves CommitLog, Consume Queue, Index File three data files. Because both memory and disk are limited resources, it is impossible for Broker to store all data permanently, so some data beyond the storage period will be deleted regularly. RocketMQ deletes additional data files by setting the data expiration time. The specific implementation logic is implemented by the periodic execution method DefaultMessageStore.this.cleanFilesPeriodically() started by the org.apache.rocketmq.store.DefaultMessageStore.start() method.
• 6.5 Broker master-slave synchronization mechanism, in the current mainstream distributed components, availability is essential. This section mainly talks about the design and implementation of usability in RocketMQ. The main content includes the following two aspects: ● Introduction to master-slave synchronization. ● Master-slave synchronization process.
• 6.6 Broker's shutdown recovery mechanism, reliability is also one of the essential features of current mainstream distributed products, especially for a financial-grade reliable message queue component. The main contents of this section are as follows: ● Related documents and implementation principles of the shutdown recovery mechanism. ● The recovery process of the shutdown recovery mechanism.

 

RocketMQ features-transaction message and delayed message mechanism;

• 7.1 Overview of transaction messages. Currently, the realization schemes of transaction messages are mainly divided into two types: two-phase commit scheme and three-phase commit scheme. RocketMQ adopts a two-phase commit scheme to implement, and the code explanation of transaction messages is based on version 4.3.0.
• 7.2 Transaction message mechanism, we summarize the sending and processing of transaction messages into four processes: Producer sends transaction messages and executes local transactions, Broker stores transaction messages, Broker checks back transaction messages, Broker commits or rolls back transaction messages. Next, we explain these four processes in detail.
• 7.3 Overview of delayed messages, what are delayed messages? Delayed messages are also called timing messages. Generally, after the producer sends the message, the consumer hopes to consume it again after a specified period of time. The conventional practice is to store the information in the database, use timed tasks to scan, and then send the eligible data to the consumer. Let's explain through a scene of buying tickets for the Spring Festival.
• 7.4 Delayed message mechanism. Before RocketMQ 4.3.0 supports delayed messages, the open source version of RocketMQ delayed message mechanism is a mystery. This section will reveal the storage and delivery mechanism of delayed messages based on RocketMQ 4.3.0.

 

RocketMQ source code reading;

• 8.1 Overview of RocketMQ source code structure. The Apache RocketMQ project is a multi-module Java project based on maven. Import the source code into IntelliJ IDEA;
• 8.2 RocketMQ source code compilation;
• 8.3 How to read the source code,
• 8.4 Source code reading example: Query messages by message id. In RocketMQ Console, we found that the content of the message body can be queried by message id. Next, let’s see how to query specifically.

 

RocketMQ enterprise best practices;

• 9.1 Overview of RocketMQ landing,
• 9.2 RocketMQ cluster management. Topic management is a common operation in cluster management, including creating topics, viewing topic routes, modifying topic configurations, and resetting consumption locations. In the previous section, I mainly explained how to manage topics, and this section mainly explained how to manage consumers. Generally, consumer management refers to consumer group management and consumer instance management. The related concepts of consumer groups and consumer instances have been described in detail in Chapter 4.
• 9.3 RocketMQ cluster monitoring and alarm, the monitoring of RocketMQ is divided into hardware monitoring and software monitoring. Hardware monitoring is generally host monitoring such as machine memory usage, CPU usage, and disk usage; software monitoring is indicator data monitoring and alarms for Namesrv, Broker, client production, and consumption. General operation and maintenance readers will add host monitoring by default. Here the author mainly introduces the software monitoring and alarm based on Prometheus.
• 9.4 RocketMQ cluster migration, RocketMQ cluster migration refers specifically to the migration of Namesrv and Broker. General corporate needs need to meet two points: transparency in R&D and no loss of information. Because the Namesrv machines are stateless, the migration is relatively simple. Here we mainly introduce how to migrate the Broker cluster.
  Broker cluster migration can be carried out by topic expansion. The author summarizes the specific migration process as
• 9.5 RocketMQ test environment practice 9.6 RocketMQ access practice, why do I need to talk about the test environment separately? The end users of the message queue middleware are the company's R&D personnel, and the test environment is also the most frequently used environment for R&D, and naturally the most problematic. The test environment has multiple sub-environments for the convenience of R&D personnel, which conflicts with the rule that the subscription relationship in RocketMQ must be consistent, leading to the "grabbing consumption" of consumers in different environments. In this section, we mainly talk about the issue of "grab consumption".

 

This [RocketMQ Distributed Messaging Middleware: Core Principles and Best Practices] PDF has 365 pages. If you need the full version, you can forward this article and follow the editor, scan the code below to get it! !

I hope that readers can be familiar with, learn from, and refer to RocketMQ's excellent design concepts in their daily work, and further their technical capabilities to better serve the company in their work.