Upload a note made by looking at the Producer source code before.
The text is as follows:
RocketMQ version: 4.5.1
The client is a separate Module, in the rocketmq/client directory.
1. Suggestion: Look at the use of Produce API from the unit test
In general unit testing, each use case is a part of the test code or a small process.
For some relatively complete open source software, their unit test coverage is very high, and it is easy to find the test cases corresponding to the process we care about. Our source code analysis can start with these test cases, follow the method call link step by step, and clarify the implementation process.
First, let's analyze the unit test of RocketMQ client to see which APIs the Producer provides, and more importantly, understand how these APIs should be used.
All the test cases of the Producer are in the same test class "org.apache.rocketmq.client.producer. DefaultMQProducerTest ". Looking at all the unit test methods in this test class, you can roughly understand the main functions of the Producer.
The main test methods of this test class are as follows:
init
terminate
testSendMessage_ZeroMessage testSendMessage_NoNameSrv testSendMessage_NoRoute testSendMessageSync_Success testSendMessageSync_WithBodyCompressed testSendMessageAsync_Success testSendMessageAsync testSendMessageAsync_BodyCompressed testSendMessageSync_SuccessWithHook
Among them, init and terminate are the codes that need to be executed when the test starts to initialize and the test ends to be destroyed. The other methods starting with testSendMessage are test cases for sending messages in various situations and scenarios. Through the names of these use cases, you can roughly See the function of the test.
For example, testSendMessageSync and testSendMessageAsync are test cases for synchronous sending and asynchronous sending, testSendMessageSync_WithBodyCompressed is a test case for compressed message sending, and so on.
Open source projects like RocketMQ spend a lot of time writing test cases in the early stage, which seems to be a waste of time. In fact, it will save a lot of time for later joint debugging testing, integration testing, and problem solving after going online, and it can effectively reduce the line. The probability of failure is generally very cost-effective. It is strongly recommended that you write more test cases in the daily development process, and try to achieve unit test coverage of more than 50%.
RockectMQ's Producer entry class is "org.apache.rocketmq.client.producer. DefaultMQProducer ". A rough look at the inheritance relationship between the code and the class, I sorted out several core classes and interfaces related to the Producer as follows:
RocketMQ uses a design pattern: facade pattern (Facade Pattern), also called appearance pattern .
The main function of the facade mode is to provide the client with an interface that can access the system and hide the internal complexity of the system.
The interface MQProducer is the facade of this mode. The client can access the Producer to realize the related functions of message sending as long as the client uses this interface. From the use level, there is no need to deal with other complex implementation classes.
The DefaultMQProducer class implements the interface MQProducer, and most of the method implementations in it do not have any business logic, but only encapsulate method calls to other implementation classes, which can also be understood as part of the facade. Most of the business logic of Producer is implemented in the class DefaultMQProducerImpl, which we will focus on later.
Sometimes, our implementation is scattered in many internal classes, and it is not convenient to use interfaces to provide external services. You can imitate RocketMQ's approach and use facade mode to hide internal implementations and provide external services.
The interface MQAdmin defines some metadata management methods, which will be used in the message sending process.
Second, the startup process
Through the code in the unit test, we can see that in the two test methods of init() and terminate() , the start and shutdown methods of Producer are executed respectively , indicating that in RocketMQ, Producer is a stateful service , which is The Producer needs to be started before the message. This startup process is actually the preparation for sending messages. Therefore, before analyzing the message sending process, we need to clarify which states are maintained in the Producer, and what initialization work the Producer does during the startup process. Only with this foundation can we analyze the implementation process of sending messages.
First start with the init() method of the test case:
@Before
public void init() throws Exception {
String producerGroupTemp = producerGroupPrefix + System.currentTimeMillis();
producer = new DefaultMQProducer(producerGroupTemp);
producer.setNamesrvAddr("127.0.0.1:9876");
producer.setCompressMsgBodyOverHowmuch(16);
// 省略构造测试消息的代码
producer.start();
// 省略用于测试构造 mock 的代码
}The logic of this initialization code is very simple. It creates an instance of DefaultMQProducer , initializes some parameters for it, and then calls the start method to start it.
Next, we follow up the implementation of the start method and continue to analyze its initialization process.
The DefaultMQProducerImpl#start() method is directly called in the DefaultMQProducer#start() method. Let’s look at the code of this method directly:
Here, RocketMQ uses a member variable serviceState to record and manage its own service state , which is actually a variant implementation of the State Pattern design pattern.
The state pattern allows an object to change its behavior when its internal state changes. The object looks like it has changed its class.
Different from the standard state mode, it does not use state subclasses, but uses switch-case to implement different behaviors in different states . When managing relatively simple states, using this design will make the code More concise. This mode is very widely used to manage stateful classes, and it is recommended that you use it in daily development.
When designing the state, there are two main points that need to be paid attention to:
The first is, not only to design a normal state , but also to design an intermediate state and the abnormal state , otherwise, once the system is abnormal, your status is not accurate, you would be very difficult to deal with this abnormal state. For example, in this code, RUNNING and SHUTDOWN_ALREADY are normal states, CREATE_JUST is an intermediate state, and START_FAILED is an abnormal state.
The second point is to consider clearly the transition paths between these states, and when performing state transitions, check whether the previous state can transition to the next state. For example, here, only in the CREATE_JUST state can it be converted to the RUNNING state, so that it can be ensured that the service is one-time and can only be started once. This avoids various problems caused by starting the service multiple times.
Next, let's take a look at the implementation of the startup process:
1. By a single-mode embodiment of MQClientManager acquired MQClientInstance of example mQClientFactory , does not automatically create a new instance;
2. register itself in the mQClientFactory;
3. Start mQClientFactory;
4. heartbeat sent to all Broker.
Here again uses one of the simplest design patterns: singleton pattern (singleton pattern involves a single class, this class is responsible for creating its own object, while ensuring that only a single object is created. This class provides a way to access its unique The method of the object can be directly accessed, without the need to instantiate the object of this class.
Examples mQClientFactory wherein corresponding class MQClientInstance is RocketMQ top level class in the client , in most cases, can be simply interpreted as corresponding to each client in an instance of class MQClientInstance. This instance maintains most of the state information of the client, as well as all the instances of Producer, Consumer and various services. Students who want to learn the overall structure of the client can start with the analysis of this class and gradually refine the analysis.
Let's further analyze the code in MQClientInstance#start():
The comments of this part of the code are relatively clear, and the process is like this:
1. Start the instance mQClientAPIImpl, where mQClientAPIImpl is an instance of the class MQClientAPIImpl, which encapsulates the communication method
between the client and the Broker; 2. Starts various timing tasks, including timing heartbeats with the Broker, timing data synchronization with NameServer and other tasks;
3 . Start the pull message service;
4. Start the Rebalance service;
5. Start the default Producer service.
The above is the startup process of Producer.
There are several important classes, and you need to be clear about their respective responsibilities. When you use RocketMQ later, if you encounter problems and need to debug your code, understanding the responsibilities of these important classes will be of great help to you:
1. DefaultMQProducerImpl: The internal implementation class of the Producer. Most of the Producer's business logic, that is, the logic of sending messages, are in this class.
2. MQClientInstance: This class encapsulates some general business logic of the client, whether it is Producer or Consumer, when you finally need to interact with the server, you need to call the methods in this class;
3. MQClientAPIImpl: This class encapsulates the client The RPC on the server side hides the specific implementation of the real network communication part from the
caller ; 4. NettyRemotingClient: The underlying implementation class of network communication between RocketMQ processes.
Three, the message sending process
Next, we analyze the process of the Producer sending messages together.
In the Producer interface MQProducer , 19 methods of sending messages with different parameters are defined, which can be divided into three categories according to the different sending methods:
- Oneway sending (Oneway): return immediately after sending the message, do not process the response, and do not care whether the sending is successful or not;
- Synchronous sending (Sync): waiting for a response after sending a message;
- Asynchronous Send (Async): Return immediately after sending the message, and handle the response in the provided callback method.
The three types of sending are basically the same, and asynchronous sending is slightly different. Let's take a look at the implementation method of asynchronous sending "DefaultMQProducerImpl#send()" (corresponding to line 1132 in the source code):
We can see that RocketMQ uses an ExecutorService to achieve asynchronous sending: use the thread pool of asyncSenderExecutor, asynchronously call the method sendSelectImpl(), and continue the follow-up work of sending messages. The current thread submits the sending task to asyncSenderExecutor and it returns. The realization of one-way sending and synchronous sending is to directly call the method sendSelectImpl() in the current thread.
Let's continue to see the implementation of the method sendSelectImpl() :
The main function of the method sendSelectImpl() is to select the queue to be sent , and then call the method sendKernelImpl() to send the message .
Which queue to choose to send is determined by the MessageQueueSelector#select method . Here RocketMQ use the strategy pattern (Strategy Pattern), to address different scenarios require different queue selection algorithm problem. (Strategy mode: define a series of algorithms, encapsulate each algorithm, and allow them to be replaced with each other. The strategy mode allows the algorithm to change independently of the customers who use it)
RocketMQ provides many implementations of MessageQueueSelector , such as random selection strategy , ha Hope to choose the strategy and the same computer room selection strategy, etc., if necessary, you can also implement the selection strategy yourself . As mentioned in our previous course, if you want to ensure the strict order of messages with the same key, you need to use a hash selection strategy, or provide a selection strategy that you implement yourself.
Next we look at the method sendKernelImpl() . This method has a lot of code, about 200 lines, but the logic is relatively simple. The main function is to construct the header RequestHeader andContext SendMessageContext , and then call the method MQClientAPIImpl#sendMessage() to send the message to the Broker where the queue is located.
At this point, the message is sent to the package class MQClientAPIImpl for remote calls to complete the subsequent serialization and network transmission steps.
It can be seen that the entire message sending process of RocketMQ Producer, whether it is synchronous or asynchronous sending, is unified into the same process . Including the realization of sending messages asynchronously, it is actually realized through a thread pool, in which calls executed in asynchronous threads and synchronous sending are the same underlying methods .
In the code of the underlying method , a parameter of the method is used to distinguish between synchronous and asynchronous sending . The advantages of this implementation are: the entire process is unified, many synchronous and asynchronous common logic, code can be reused, and the code structure is clear and simple, which is easy to maintain .
- When using synchronous sending , the current thread will block waiting for the response from the server, and will not return until the response is received or the timeout method is received, so when the business code calls synchronous sending, as long as the return is successful, the message must be sent successfully.
- When sending asynchronously , the sending logic is executed in the asynchronous thread of the Executor, so the current thread will not be blocked. When the server returns a response or times out, the Producer will call the Callback method to return the result to the business code. The business code needs to judge the sending result in the Callback.
to sum up:
The realization process of RocketMQ client message production, including the main process of Producer initialization and message sending.
Several core services included in the Producer are all stateful . When the Producer starts, they are started in the MQClientInstance class.
In the process of sending messages, RocketMQ is divided into three sending methods: one-way , synchronous and asynchronous . The sending processes corresponding to these three sending methods are basically the same . Synchronous and asynchronous sending are separated by the encapsulated MQClientAPIImpl class . Achieved.
For the several important business logic implementation classes we mentioned in the analysis code, you'd better remember these classes and their functions, including: DefaultMQProducerImpl encapsulates most of the Producer’s business logic, and MQClientInstance encapsulates the client Some general business logic, MQClientAPIImpl encapsulates the RPC between the client and the server, and NettyRemotingClient implements the underlying network communication.
Of course, this is just the main process, detailed details, and then look at the source code.