Detailed explanation of RocketMQ (http://www.wfuyu.com/mvc/25852.html)

Remarks:
1. If you have not contacted RocketMQ before, please browse the appendix first to understand the overall architecture and related terms
of RocketMQ 2. MQServer and Broker in the text represent the same concept.
Distributed messaging system can be expanded as a distributed system. , the key components of scalability, need to have the characteristics of high throughput and high availability. When it comes to the design of the message system, two problems cannot be avoided:
the order of
messages and the repetition of messages.
RocketMQ is a high-performance and high-throughput message middleware open sourced by Alibaba. How does it solve these two problems? of? What are the key features of RocketMQ? What is the principle of its realization?
Key features and implementation principles
1. Sequential messages
Message ordering means that when type 1 messages are consumed, they can be consumed in the order in which they are sent. For example: 1 order generates 3 messages, namely order creation, order payment, and order completion. When consuming, it only makes sense to consume according to this order. But at the same time, the orders can be consumed in parallel.
Suppose the producer generates two messages: M1 and M2, what should be done to ensure the order of these two messages? What you think of in your mind is this:
write a picture here to describe
that after M1 is sent to S1, M2 is sent to S2. If you want to ensure that M1 is consumed before M2, you need to notify S2 after M1 reaches the consumer, and then S2 will send M2 to M2. sent to the consumer.
The problem with this model is that if M1 and M2 are sent to two servers respectively, it cannot guarantee that M1 arrives first, and it cannot guarantee that M1 is consumed first, so the order of messages needs to be protected in the MQ Server cluster. Then how to solve it? A simple way is to send M1 and M2 to the same server:
write the picture description here
This ensures that M1 arrives at the MQServer before M2 (the client waits for M1 to succeed before sending M2). According to the principle of "first arrival and first consumption", M1 will be consumed before M2, thus ensuring the order of messages.
This model can theoretically guarantee the order of messages, but in practical applications you should encounter the following problems:
write picture description here
As long as messages are sent from one server to another server, there will be a network delay problem. As shown in the figure above, if sending M1 takes longer than sending M2, then M2 will be consumed first, and the order of messages cannot be guaranteed. Even if M1 and M2 arrive at the consumer end at the same time, because the load conditions of consumer end 1 and consumer end 2 are not clear, M2 may still be consumed before M1. how to solve this problem? It is enough to send M1 and M2 to the same consumer, and after sending M1, M2 can only be sent after the consumer responds successfully.
But another problem will be introduced. If the consumer 1 does not respond after sending M1, should it continue to send M2 or resend M1? 1 Generally, in order to ensure that message 1 will be consumed, it will definitely choose to resend M1 to another 1 consumer 2, as shown in the figure below.
Write a picture description here
This model strictly guarantees the order of messages. If you are careful, you will still find problems. There are two situations when consumer 1 does not respond to the server. One is that M1 does not arrive, and the other is consumer 1. It has responded, but the server side has not received it. If it is the second case, retransmitting M1 will cause M1 to be repeatedly consumed. That is the second problem we will talk about later, the problem of message duplication.
Looking back at the problem of message order, strict sequential messages are very easy to understand and easier to deal with. To achieve strict sequential messages, the simple and feasible way is to
ensure that the producer-MQServer-consumer is a 1-to-1 pair 1 relationship
However, with this design, the degree of parallelism becomes the bottleneck of the message system (insufficient throughput), and it will also lead to more exception handling. For example, as long as there is a problem on the consumer side, it will cause the entire processing process to be blocked. We have to spend more energy to solve the blocking problem.
But our ultimate goal is high fault tolerance and high throughput of the cluster. This seems to be a pair of irreconcilable contradictions, so how did Ali solve it?
The easiest way in the world to solve 1 computer problem: "just" don't need to solve it! —— Shen Xun
Some problems seem to be very important, but in fact we can avoid them through reasonable design or decomposition of problems. It is actually wasted and ineffective to spend time trying to fix them. Looking at the order of messages from this perspective, we can draw two conclusions:
1. There are actually a lot of uses that do not pay attention to disorder
. 2. Disordered queues do not actually mean disordered messages.
Finally we analyze how RocketMQ is implemented from the perspective of source code. Send sequential messages.
1. Generally, messages are sent by polling all queues (load balancing strategy), and sequential messages can be sent to the same queue according to the business, such as messages with the same order number. In the following example, messages with the same OrderId will be sent to the same queue:
// RocketMQ provides two MessageQueueSelector implementations by default: random/Hash SendResult sendResult = producer.send(msg, new MessageQueueSelector() { @Override public MessageQueue select(List<MessageQueue> mqs, Message msg, Object arg) { Integer id = (Integer) arg; int index = id % mqs.size(); return mqs.get(index); } }, orderId);
After the routing information is obtained, a queue will be selected according to the algorithm implemented by MessageQueueSelector, and the queue obtained with the same OrderId is the same queue.
private SendResult send() { // Get topic routing information TopicPublishInfo topicPublishInfo = this.tryToFindTopicPublishInfo(msg.getTopic()); if (topicPublishInfo != null && topicPublishInfo.ok()) { MessageQueue mq = null; // According to our Algorithm, select 1 send queue // here arg = orderId mq = selector.select(topicPublishInfo.getMessageQueueList(), msg, arg); if (mq != null) { return this.sendKernelImpl(msg, mq, communicationMode, sendCallback, timeout); } } }
2. Message repetition When solving the problem of message sequence
above , a new problem was introduced, that is, message repetition. So how does RocketMQ solve the problem of message duplication? Still "just" doesn't solve it.
The fundamental reason for the duplication of messages is that the network is unreachable. As long as data is exchanged over the network, there is no way to avoid this problem. So the solution to this problem is not to solve it, but to bypass it. Then the question becomes: if the consumer receives two identical messages, what should they do?
1. The business logic of processing messages on the consumer side remains idempotent
2. Ensure that each message has a unique number and ensure that the message processing is successful and the log of the deduplication table appears at the same time
Item 1 is easy to understand, as long as idempotency is maintained, no matter how many duplicate messages come, the final result will be the same. The second principle is to use a log table to record the ID of the successfully processed message. If the newly arrived message ID is already in the log table, then this message will not be processed.
We can see the solution of Article 1. Obviously, it should be implemented on the consumer side, and it is not a function to be implemented by the message system. Item 2 can be implemented by the message system or by the business side. Under normal circumstances, the probability of duplicate messages is not necessarily large, and if it is implemented by the message system, it will definitely affect the throughput and high availability of the message system. Therefore, it is best for the business side to handle the problem of message duplication by itself. The reason why RocketMQ does not solve the problem of message duplication.
RocketMQ does not guarantee that messages are not repeated. If your business needs to ensure strict non-duplication of messages, you need to deduplicate them on the business side.
3. Transaction messages
RocketMQ supports transaction messages in addition to ordinary messages and sequential messages. First discuss what is a transaction message and the necessity of supporting transaction messages. Let's take a transfer scenario as an example to illustrate this problem: Bob transfers 100 blocks to Smith.
In a stand-alone environment, the situation of executing a transaction is probably as follows:
Write a picture description here
When the number of users grows to a certain level, the account and balance information of Bob and Smith are no longer on the same server, then the above process will be It has become like this:
write a picture description here. At
this time, you will find that the same transaction is a transfer, but in a cluster environment, the time-consuming increases exponentially, which is obviously unacceptable. So how do we circumvent this problem?
Large transaction = small transaction +
asynchrony Split large transaction into multiple small transactions for asynchronous execution. In this way, the execution efficiency of the cross-machine transaction can be optimized to be consistent with the single machine 1. The transaction of transfer can be decomposed into the following two small transactions:
write a picture description here
In the figure, performing local transactions (Bob account deduction) and sending asynchronous messages should remain successful or unsuccessful at the same time, that is, if the deduction is successful, sending message 1 must succeed. If the deduction fails, no more messages can be sent. The question is: do we charge first or send messages first?
First of all, let's take a look and send the message first. The general diagram is as follows:
write the picture description
here . The problem is: if the message is sent successfully, but the deduction fails, the consumer will consume the message and then add money to the Smith account.
If you can’t send the message first, then we will deduct the money first. The general diagram is as follows:
write a picture description here. The
existing problems are similar to the above: if the deduction is successful and the message fails to be sent, Bob will deduct the money, but Smith’s account has not been deducted. Add money.
There may be many ways to solve this problem, such as: directly send the message to the transaction of Bob's deduction, if the sending fails, an exception is thrown and the transaction is rolled back. This way of handling is also in line with the principle that "just" does not need to be resolved. RocketMQ supports transaction messages. Let's take a look at how RocketMQ implements it.
Write a picture here to describe that
when RocketMQ sends a Prepared message in the first stage, it will get the address of the message, the second stage performs local transactions, and the third stage accesses the message through the address obtained in the first stage and modifies the state. Carefully, you may find the problem again. What if the confirmation message fails to be sent? RocketMQ will regularly scan the transaction messages in the message cluster. At this time, when a Prepared message is found, it will confirm to the message sender whether Bob's money has been reduced or not? If the decrease is to roll back or continue to send a confirmation message? RocketMQ will decide whether to roll back or continue to send confirmation messages according to the policy set by the sender. This ensures that the message sending succeeds or fails at the same time as the local transaction.
Then let's take a look at the RocketMQ source code to see if this is how transaction messages are processed. The part where the client sends a transaction message (for the complete code, please see: com.alibaba.rocketmq.example.transaction.TransactionProducer under the rocketmq-example project):
// For pending transactions, the MQ server checks the client back // That is, the above As mentioned in the article, when RocketMQ finds a `Prepared message`, it will determine the transaction according to the strategy implemented by this Listener transactionCheckListener = new TransactionCheckListenerImpl(); // Construct the producer of transaction messages TransactionMQProducer producer = new TransactionMQProducer("groupName") ; // Set the transaction decision processing class producer.setTransactionCheckListener(transactionCheckListener); // The processing logic of local transactions is equivalent to the logic of checking Bob's account and deducting money in the example TransactionExecuterImpl tranExecuter = new TransactionExecuterImpl(); producer.start() // Construct MSG, omit construction parameters Message msg = new Message(...); // Send message SendResult sendResult = producer.sendMessageInTransaction(msg, tranExecuter, null); producer.shutdown();
Then look at the source code of the sendMessageInTransaction method, which is divided into three stages: sending Prepared messages, performing local transactions, and sending confirmation messages.
public TransactionSendResult sendMessageInTransaction(.....) { // logic code, not actual code // 1. Send message sendResult = this.send(msg); // sendResult.getSendStatus() == SEND_OK // 2. If message Send successfully, process the local transaction unit LocalTransactionState associated with the message localTransactionState = tranExecuter.executeLocalTransactionBranch(msg, arg); // 3. End the transaction this.endTransaction(sendResult, localTransactionState, localException); } The
endTransaction method will send the request to the broker( mq server) to update the final status of the transaction message:
1. Find the Prepared message
according to sendResult 2. Update the final status of the message according to localTransaction
If the endTransaction method fails and the data is not sent to the broker, the broker will have a check-back thread periodically (1 minute by default) to scan each table file that stores the transaction status. If it is a submitted or rolled back message, skip it directly. The prepared state will initiate a CheckTransaction request to the Producer, the Producer will call the DefaultMQProducerImpl.checkTransactionState() method to process the broker's timing callback request, and checkTransactionState will call the decision method of our transaction settings, and finally call endTransactionOneway to let the broker update the final state of the message .
Going back to the transfer example, if the balance of Bob's account has been reduced and the message has been sent successfully, the Smith side starts to consume the message. At this time, there will be two problems of consumption failure and consumption timeout? The idea to solve the timeout problem is to retry until the consumer end consumes the message successfully. There may be a problem of message duplication in the whole process, which can be solved according to the previous idea.
Write a picture description here
This basically solves the timeout problem, but what if the consumption fails? The solution provided by Ali is: manual solution. You can think about it, according to the business process, if Smith fails to add funds for some reason, you need to roll back the entire process. If the message system wants to implement this rollback process, the system complexity will be greatly increased, and bugs are prone to occur. It is estimated that the probability of bugs will be much greater than the probability of consumption failures. We need to measure whether it is worth paying such a large price to solve such a problem with a very small probability, which is also where everyone needs to think more when solving doubtful problems.
20160321 Added: The implementation of transaction messages was removed in version 3.2.6, so this version does not support transaction messages. For details, please refer to rocketmq's issues:
https://github.com/alibaba/RocketMQ/issues/65
https://github.com/alibaba/RocketMQ/issues/138
https://github.com/alibaba/RocketMQ/issues/156
4. How does the Producer send messages?
The Producer polls all queues under a topic to achieve sending The load balancing of the party, as shown in the following figure:
Write the picture description here
First analyze the source code of the message sent by the RocketMQ client:
// Construct the Producer DefaultMQProducer producer = new DefaultMQProducer("ProducerGroupName"); // Initialize the Producer, all use life In the cycle, you only need to initialize once producer.start(); // Construct Message Message msg = new Message("TopicTest1",// topic "TagA",// tag: tag the message to identify type 1 message , can be null "OrderID188",// key: custom Key, can be used for deduplication, can be null ("Hello MetaQ").getBytes());// body: message content // Send message and return result SendResult sendResult = producer.send(msg); // Liquidate resources, close network connections, and log off producer.shutdown();
During the entire utilization life cycle, the producer needs to call the start method once to initialize and initialize the main completed tasks Yes :
1. If no namesrv address is specified, it will be addressed automatically
2. Start timing tasks: update the namesrv address, update topic routing information from namsrv, clear the brokers that have been suspended, send heartbeats to all brokers...
3. Start the load balancing service After the
initialization complete, start sending messages, the main code for sending messages The following:
private SendResult sendDefaultImpl(Message msg,...) { // Check whether the state of the Producer is RUNNING this.makeSureStateOK(); // Check whether msg is legal: is it null, topic, body is Is not empty, body is too long Validators.checkMessage(msg, this.defaultMQProducer); // Get topic routing information TopicPublishInfo topicPublishInfo = this.tryToFindTopicPublishInfo(msg.getTopic()); // Select 1 message from routing information Queue MessageQueue mq = topicPublishInfo.selectOneMessageQueue(lastBrokerName); // Send message to this queue sendResult = this.sendKernelImpl(msg, mq, communicationMode, sendCallback, timeout); }
The two methods that need attention in the code are tryToFindTopicPublishInfo and selectOneMessageQueue. As mentioned earlier, when the producer is initialized, it will start a scheduled task to obtain routing information and update it to the local cache, so tryToFindTopicPublishInfo will first obtain topic routing information from the cache. If it is not obtained, it will go to namesrv to obtain routing information. The selectOneMessageQueue method returns a queue by polling to achieve the purpose of load balancing.
If the Producer fails to send the message, it will automatically retry. The retry strategy:
the number of retries < retryTimesWhenSendFailed (configurable)
the total time (including the time required to retry n times) < sendMsgTimeout (parameters passed in when sending the message)
After the above two conditions are met at the same time, the Producer will select another queue to send messages
. 5. Message storage
RocketMQ's message storage is completed by the cooperation of consume queue and commit log.
1. Consume Queue
The consume queue is a logical queue of messages, equivalent to a directory of dictionaries, used to specify the location of messages on the physical file commit log.
We can specify the directory where consumequeue and commitlog are stored in the configuration
. Each queue under each topic has a corresponding consumequeue file, for example:
${rocketmq.home}/store/consumequeue/${topicName}/${queueId }/${fileName}
Consume Queue file organization, as shown in the figure:
Write the picture description here to organize files
according to topic and queueId. In the figure, TopicA has two queues 0 and 1. Then TopicA and QueueId=0 form one ConsumeQueue, and TopicA and QueueId=1 form another ConsumeQueue.
The retry queue is grouped according to the actual GroupName of the consumer. If the consumer fails to consume, the message will be sent to the retry queue, such as %RETRY%ConsumerGroupA in the figure.
Group dead letter queues according to the actual GroupName of the consumer. If the consumer fails to consume and retries the specified number of times and still fails, it will be sent to the dead letter queue, such as %DLQ%ConsumerGroupA in the figure.
Dead Letter Queue 1 is generally used to store messages that cannot be delivered for some reason, such as processing failed or expired messages.
The storage unit in the Consume Queue is a 20-byte fixed-length binary data, which is written and read sequentially, as shown in the following figure:
Write picture description here
CommitLog Offset refers to the actual offset of this message in the Commit Log file
The size of the message in the Size storage
Message Tag HashCode stores the hash value of the Tag of the message: mainly used for message filtering when subscribing (if a Tag is specified when subscribing, the subscribed message will be quickly found according to the HashCode)
2. Commit Log
CommitLog: The physical file for message storage. The commitlog on each broker is shared by all the queues of the local machine without any distinction.
The default location of the file is as follows, which can still be modified through the configuration file:
${user.home} \store\${commitlog}\${fileName}
The length of the message storage unit of CommitLog is not fixed, and the files are written sequentially and read randomly. The storage structure of the message is shown in the following table, which is stored in the order of the number and the content corresponding to the number.
Write a picture description here
3. Message storage
implementation The implementation of message storage is relatively complicated, and it is worth your in-depth understanding. It will be analyzed in a separate document later. This section only uses code to explain the specific process.
// Set the storage time msg.setStoreTimestamp(System.currentTimeMillis()); // Set the message body BODY CRC (consider the most appropriate setting msg.setBodyCRC(UtilAll.crc32(msg.getBody())); StoreStatsService storeStatsService = this.defaultMessageStore.getStoreStatsService(); synchronized (this) { long beginLockTimestamp = this.defaultMessageStore.getSystemClock().now(); // Here settings are stored timestamp, in order to ensure an orderly global msg.setStoreTimestamp(beginLockTimestamp); // MapedFile：操作物理文件在内存中的映照和将内存数据持久化到物理文件中 MapedFile mapedFile = this.mapedFileQueue.getLastMapedFile(); // 将Message追加到文件commitlog result = mapedFile.appendMessage(msg, this.appendMessageCallback); switch (result.getStatus()) { case PUT_OK:break; case END_OF_FILE: // Create a new file, re-write the message mapedFile = this.mapedFileQueue.getLastMapedFile();result = mapedFile.appendMessage(msg, this.appendMessageCallback); break; DispatchRequest dispatchRequest = new DispatchRequest( topic,// 1 queueId,// 2 result.getWroteOffset(),// 3 result.getWroteBytes(),// 4 tagsCode,// 5 msg.getStoreTimestamp(),// 6 result.getLogicsOffset(),// 7 msg.getKeys(),// 8 /** * Transaction */ msg.getSysFlag(),// 9 msg.getPreparedTransactionOffset());// 10 // 1.分发消息位置到ConsumeQueue // 2.分发到IndexService建立索引 this.defaultMessageStore.putDispatchRequest(dispatchRequest); }getPreparedTransactionOffset());// 10 // 1. Distribute message location to ConsumeQueue // 2. Distribute to IndexService for indexing this.defaultMessageStore.putDispatchRequest(dispatchRequest); }getPreparedTransactionOffset());// 10 // 1. Distribute message location to ConsumeQueue // 2. Distribute to IndexService for indexing this.defaultMessageStore.putDispatchRequest(dispatchRequest); }
4. The index file of the message
If a message includes a key value, the index file will be used to store the message index. The content structure of the file is as follows:
write a picture description here. The
index file is mainly used to query messages according to the key. The main process is:
1 . Obtain the specific slot position according to the hashcode%slotNum of the queried key (slotNum is the maximum number of slots included in an index file, for example, slotNum=5000000 as shown in the figure)
2. According to slotValue (the value corresponding to the slot position) Find the last item of the index item list (reverse order, slotValue always points to the latest index item)
3. Traverse the index item list and return the result set within the query time range (by default, a maximum of 32 records are returned)
6 , Message
subscription RocketMQ message subscription has two modes, one is Push mode, that is, MQServer actively pushes to the consumer; the other is Pull mode, that is, the consumer actively pulls from MQServer when needed. However, in the specific implementation, the Push and Pull modes are both actively pulled by the consumer.
First, let's take a look at the actual load balancing of consumption:
write the picture description here. The
consumer will use the RebalanceService thread to do all the queue loads based on the topic once every 10 seconds:
1. Traverse all the topics under the Consumer, and then subscribe to all the topics according to the topic. Message
2. Obtain all Consumers under the same topic and Consumer Group.
3. Then allocate consumption queues according to specific allocation strategies. The allocation strategies include: average allocation, consumer configuration, etc.
As shown in the figure above: if there are 5 queues and 2 consumers, then the first consumer consumes 3 queues, and the second consumer consumes 2 queues. The average distribution strategy adopted here is similar to our paging. All queues under TOPIC are records, and the number of consumers is equivalent to the total number of pages. How many records per page is similar to a Consumer meeting Which queues to consume.
Through such a strategy to achieve a general average consumption, such a design can also expand the consumer to a high level to improve the consumption capacity.
The consumption of the push mode is realized by the mode of too long polling, just like the following figure:
write a picture here to describe that the consumer
side actively sends a pull message request to the broker every 1 period. After the broker receives the Pull request, if When there is a message, it returns the data immediately. After the Consumer receives the returned message, it calls back the Listener method set by the consumer. If there is no data in the message queue when the broker receives a Pull request, the broker side will block the request until there is data delivery or timeout before returning.
Of course, the Consumer side sends the PullRequest in the blocking queue LinkedBlockingQueue to the broker to pull the message through one thread to prevent Consumer1 from being blocked. On the Broker side, when it receives the Consumer's PullRequest, if it finds that there is no message, it will throw the PullRequest into the ConcurrentHashMap and cache it. When the broker starts, it will start a thread to continuously check the PullRequest from the ConcurrentHashMap until data is returned.
7. Other features of
RocketMQ 6 features are basically all point to point. If you want to know more about it, you need to check the source code a lot and apply it in practice. Of course in addition to the features already mentioned, RocketMQ also supports:
1. Timed messages
2. Message flushing strategy 3.
Active synchronization strategy: synchronous double write, asynchronous replication
4. Mass message accumulation capability
5. Efficient communication
.
We conduct in-depth research:
1. Message storage design: It is necessary to meet the accumulation ability of massive messages, to meet the extremely fast query efficiency, and to ensure the efficiency of writing.
2. Efficient communication component design: high throughput and millisecond-level message delivery capability are inseparable from efficient communication.
3. .......
RocketMQ best practice
1. Producer best practice
1. Use one topic as much as possible for one application, and the message subtype is identified by tags, which can be freely set by the application. Only when tags are set for sending messages, consumers can use tags to filter messages on the broker when subscribing to messages.
2. The unique identification code of each message at the business level should be set to the keys field to facilitate locating the problem of message loss in the future. Because of this hash index, make sure that the key is as unique as possible to avoid potential hash collisions.
3. If the message is sent successfully or failed, to print the message log, be sure to print the sendresult and key fields.
4. The message cannot be lost and used, and there must be a message retransmission mechanism. For example, if the message fails to be sent, it is stored in the database, and there can be a timed program to try to retransmit or manually trigger the retransmission.
5. For some applications, if you do not pay attention to whether the message is sent successfully, please use the sendOneWay method to send the message directly.
2. The best practice of Consumer The
consumption process should be idempotent (that is, deduplication on the consumer side)
Use the batch consumption method as much as possible, which can greatly improve the consumption throughput
. Optimize the consumption process
of each message. 3. Other configurations AutoCreateTopicEnable should be turned off
online , that is, set it to false in the configuration file.
When RocketMQ sends a message, it will first obtain routing information. If it is a new message, since the corresponding Topic has not been created on MQServer, at this time, if the above configuration is enabled, it will return the default TOPIC (RocketMQ will create a TOPIC named TBW102 on each broker) routing information, and then The Producer will select one Broker to send the message. When the selected broker is storing the message, it will automatically create the topic when it finds that the topic of the message has not been created. The consequence is: in the future, all the messages of the TOPIC will be sent to this broker, which cannot achieve the purpose of load balancing.
Therefore, based on the current design of RocketMQ, it is recommended to turn off the function of automatically creating TOPIC, and then manually create TOPIC according to the size of the message volume.
RocketMQ design coherent
RocketMQ design assumptions:
each PC machine may be down and unserviceable
Any cluster may have insufficient processing power
Worst case 1 will definitely
occur Intranet environment requires low latency to provide the best user experience
Key design of RocketMQ :
distributed clustering,
strong data security,
massive data accumulation,
millisecond -level delivery delay (push-pull mode)
This is RocketMQ's design assumptions and desired effects. I think these assumptions apply to all system designs. With the increase in the services of our system, each developer must pay attention to whether there is a single point of failure in his program, how to restore it if it hangs, whether it can be expanded to a good level, and whether the external interface is efficient enough? Is the managed data safe enough... Only by standardizing your own design can you develop an efficient and tough program.
Appendix: Introduction of several professional terms and overall architecture touched by
RocketMQ 1. The professional term
Topic
topic in RocketMQ represents the first-level type of message. For example, the message of an e-commerce system can be divided into: transaction message, logistics message... 1 message must have 1 topic.
Tag
Tag represents the second-level type of message. For example, transaction messages can be divided into: transaction creation messages, transaction completion messages... 1 message can have no Tag. RocketMQ provides 2-level message classification, which is convenient for everyone to control flexibly.
Queue
1 topic, we can set up multiple queues (message queues). When we send a message, we need to specify the topic of the message. RocketMQ will poll all queues under the topic and send messages out.
Producer and Producer Group
Producer represents the producer of the message queue. The essence of the message queue is to implement the publish-subscribe mode, where producers produce messages and consumers consume messages. Therefore, the Producer here is used to produce and send messages, generally referring to the business system.
Producer Group is the collective name of Type 1 Producer. This type of Producer usually sends Type 1 messages, and the sending logic is consistent.
Consumer and Consumer Group
Message consumers generally consume messages asynchronously by the background system.
One type of Push Consumer
Consumer, which usually uses a Listener interface to be registered with the Consumer object. Once a message is received, the Consumer object immediately calls back the Listener interface method.
One type of Pull Consumer
Consumer, which uses the pull message method that usually calls the Consumer actively to pull messages from the Broker, and the initiative is controlled by the application.
Consumer Group is the name of a collection of Type 1 Consumers. This type of Consumer usually consumes Type 1 messages, and the consumption logic is consistent.
Broker
message relay, responsible for storing and forwarding messages. It can be understood as a message queue server, which provides services for receiving, storing, pulling and forwarding messages. The broker is the core of RocketMQ, it cannot hang, so it is necessary to ensure the high availability of the broker.
Broadcast consumption
A message is consumed by multiple consumers. Even if these consumers belong to the same Consumer Group, the message will be consumed once by each Consumer in the Consumer Group. The Consumer Group concept in broadcast consumption can be considered meaningless in terms of message partitioning.
The cluster consumes
the Consumer instances in one Consumer Group and distributes the consumption messages equally. For example, a topic has 9 messages, of which 1 Consumer Group has 3 instances (mostly 3 processes, or 3 machines), then each instance only consumes 3 of the messages.
NameServer
NameServer is the name service, with two functions:
Receive the broker's request, register the broker's routing information
Interface client's request, obtain its routing information to the broker according to a topic
NameServer has no status and can be expanded horizontally. Each broker will register with the NameServer when it is started; the Producer will obtain the routing (to the broker) information from the NameServer according to the topic before sending the message; the Consumer will also obtain the topic routing information regularly.
2. RocketMQ Overview
Write pictures here to describe
the Producer sending messages to several queues in turn. The set of queues is called Topic. If the Consumer does broadcast consumption, one consumer instance consumes all the queues corresponding to this Topic; if it does cluster consumption, multiple The Consumer instance consumes the queue set corresponding to this topic on average.
Let's look at the RocketMQ physical deployment structure diagram:
Write a picture here to describe
the characteristics of RocketMQ network deployment:
Name Server is a nearly stateless node, which can be deployed in a cluster, and there is no information synchronization between nodes.
Broker deployment is relatively complicated. Brokers are divided into Master and Slave. One Master can correspond to multiple Slaves, but one Slave can only correspond to one Master. The correspondence between Master and Slave is defined by specifying the same BrokerName and different BrokerId. BrokerId=0 means Master, non-0 means Slave. Master can also deploy multiple. Each Broker establishes long-term connections with all nodes in the Name Server cluster, and regularly registers topic information to all Name Servers.
The Producer establishes a persistent connection with one of the nodes (randomly selected) in the Name Server cluster, periodically obtains topic routing information from the Name Server, establishes a persistent connection to the Master that provides Topic services, and sends heartbeats to the Master regularly. Producers are completely stateless and can be deployed in clusters.
The Consumer establishes a persistent connection with one of the nodes (randomly selected) in the Name Server cluster, periodically obtains topic routing information from the Name Server, establishes a persistent connection to the Master and Slave that provide Topic services, and regularly sends heartbeats to the Master and Slave. Consumers can subscribe to messages from both the Master and the Slave. The subscription rules are determined by the Broker configuration.