In fact, sequential messaging is one of the commonly used functions in business. As long as MQ is used, it is unlikely to get around this problem. Even if there is no need to guarantee the sequence of messages in business, as a developer, as a coder, you have to confirm the requirements with your PM no matter what, then you will definitely talk about the sequence of messages in MQ!
To give a simple example, let’s not talk about the old-fashioned shopping transaction system, order system, etc., such as the BinLog message of the database, and the database executes new, modified, and deleted statements, so the BinLog records must also be added, modified , delete statement, this is easy to understand.
Without further ado, let’s get straight to the point~
The wind is beautiful, the sky is clear and the air is clear, and it is another suitable day for interviews. As an interviewer who has been offered nine times out of ten, I came to visit the interviewer again.
Boy, last time we talked about the performance optimization of RocketMQ, you answered well. Then do you know how RocketMQ ensures sequential messages?
Oops, isn't this insulting? I've already talked about RocketMQ myself. I'm afraid you'll ask? Then I'm afraid you won't ask.
He pushed up his glasses and pretended to think:
It’s such an interviewer boss. Sequential messages are divided into sequential sending and sequential receiving. In RocketMQ, it supports sequential messages, so it is very simple to implement in code.
Send messages sequentially:
-
First, we can change the configuration in our spring.properties to the following configuration to ensure synchronous sending:
spring.cloud.stream.rocketmq.bindings.output.pruducer.sync=true -
Secondly, when we send a message in the business code, we specify the Header to send the message to and specify it to be sent to the 0th message queue:
ThisCopy code
@RestController public class OrderlyController { @Autowired private Source source; @GetMapping(value = "/orderly") public String orderlySend() { List<String> messageList = Arrays.asList("insert", "update", "delete"); for (String message : messageList) { MessageBuilder builder = MessageBuilder.withPayload(message).setHeader(BindingHeaders.PARTITION_HEADER, 0); Message msg = builder.build(); source.output().send(message); } return "success"; } }
Receive messages sequentially:
-
Similarly, we also need to change the configuration file in spring.properties to the following configuration to ensure synchronous reception:
spring.cloud.stream.rocketmq.bindings.input.consumer.orderly=true -
When we consume messages, we only need to consume messages normally. As for the code, just write it casually:
typescriptCopy code
@StreamListener(value = Sink.INPUT) public void receive(String receivedMessage) { System.out.println(receivedMessage); }
Principle of sending messages sequentially:
It's not difficult to talk about this. Boy, do you know the principle of this sequential message? How does RocketMQ implement sequential messages?
Ahem. . . Why do you think I stopped talking in the middle of my sentence? Now, didn’t you just jump in? At this time, I knew the answer and what to do. I was very anxious and wanted to tell it all at once, but! I know it's urgent, but I don't have to be impatient and wait for him for three to five seconds.
As for RocketMQ, its sequential messages are actually divided into two types. The first is local ordering, which is the one I mentioned earlier, and the second is global ordering.
So what is local ordering and what is global ordering?
Locally ordered:
good question!
Partially ordered means that the messages sent to the same message queue are in order. You can specify the message queue to be sent when sending the message (refer to sending sequential messages above). Similarly, when consuming messages, they are also consumed in order. For example, the database operations of the same user need to be in order, and the database operations of different users have no constraints, will not affect each other, and can be parallel.
Globally ordered:
After understanding the previous one, this one is easier to understand. We only need to set the topic to have only one queue to achieve global order (have you spit out a mouthful of old blood?), we need to manually set it when creating the topic Only one queue is created. However, this type of scenario is relatively rare in China. Because of the large domestic user base, this low-performance method is usually not recommended.
At the same time, there are three ways to send messages in RocketMQ: synchronous, asynchronous, and one-way.
Synchronous sending: After sending a network request, it will synchronously wait for the return result from the Broker server. It supports retry after sending failure and is suitable for more important message notification business scenarios.
Asynchronous sending: Sending network requests asynchronously will not block the current thread, does not support retrying on failure, and is suitable for business scenarios that require relatively high response time.
One-way sending: The principle of one-way sending is the same as asynchronous, but callbacks are not supported. It is suitable for scenarios with very short response time and low reliability, such as logs.
In fact, the principle of sending messages sequentially is very simple. You only need to send the same type of messages to the same queue. In order to ensure that the message sent first arrives at the message queue first, the synchronous message sending mode must be used, otherwise the message sent later may arrive at the message queue first, and in this case, the messages will be out of order.
Here we also take a look at the source code of RocketMQ and analyze it:
The process of selecting the queue is completed by MessageQueueSelector and hashKey in the implementation class SelectMessageQueueByHash:
- Calculate the hash value based on hashKey, which is the unique key of the user in our previous example, such as user ID, so the calculated hash value is exactly the same.
- Use the hash value and the number of queues mqs.size() modulo to get an index value. The result is of course smaller than the number of queues.
- Get a queue from the queue list based on the index value msq.get(value). If the hash values are the same, the queues are the same.
In the process of obtaining the queue list, the Producer queries the Broker list from the NameServer according to the Topic, and caches it in the local memory so that it can be read from the cache next time.
Yo, you're a good guy. You can answer this. Let me ask you again, are there other types of messages in RocketMQ besides sequential messages?
Forehead. . Of course, in addition to sequential messages, RocketMQ also supports transaction messages and delayed messages. Apart from these three, the others can be called ordinary messages. The most commonly used scenarios in daily development are also ordinary messages. This is also because the most commonly used scenarios are asynchronous decoupling between systems and peak-shaving and valley-filling. Therefore, in these scenarios, the priority is to ensure the efficient sending and receiving of messages.
Comparing ordinary messages and sequential messages, ordinary messages have different strategies for selecting message queues when sending. There are two main mechanisms for selecting queues for ordinary message sending. Here is a brief introduction to polling:
-
Polling (default): As the default selection mechanism, it is also easy to understand. A Topic has multiple queues, and polling is enough to select one of them. The principle is that a counter is maintained in the routing information TopicPublishInfo. Every time it is sent, the routing is queried and the counter is +1. The polling algorithm is implemented by taking the modulo calculation of the counter value index and the number of queues.
Although the polling mechanism is simple and easy to use, it has a drawback. If the queue selected for polling is on a downed Broker, it will cause the message to fail to be sent. Even if the queue is reselected when the message is resent, it may still be selected on the downed Broker. On the Broker, there is no way to avoid sending failures, so there is a fault avoidance mechanism. We will talk about this later.
Principle of sequential consumption of messages:
You have said so much, but how is sequential consumption of messages implemented?
Oh, the problem is like this. RocketMQ supports two message modes, cluster consumption and broadcast consumption. The difference between the two is that in the broadcast consumption mode, each message will be consumed by each Comsumer in the ConsumerGroup, and in the cluster consumption mode, each message will only be consumed by one Consumer in the ConsumerGroup.
In fact, cluster consumption is used in most scenarios. A message that is not consumed once represents a business processing. This is not difficult to understand, as each message is processed by a single service instance in cluster mode. Broadcast consumption is used in a few scenarios. For example, if data changes, each service instance that receives this message needs to clear the cache. This requires every consumer in the entire cluster to consume the message once. By default, it is cluster consumption, and we will analyze sequential consumption under this condition.
Sequential consumption is also called ordered consumption. The principle is that the same message queue only allows one consumer thread in the Consumer to consume. There are multiple consumer threads in Consumer, and multiple threads will consume messages at the same time. Under this condition, it is easy to think of exclusive locks. Yes, in sequential consumption, the consumer thread will first apply for an exclusive lock, and then it will be locked after obtaining the lock. Consumption is allowed, let’s take a look at the source code:
After the consumption is successful, the consumption progress will be submitted to the Broker and the consumption location information will be updated to avoid fetching already consumed messages next time. During sequential consumption, if the consuming thread throws an exception while performing business processing in the listener, the consumption progress will not be submitted. The consumption progress will be blocked on the current message and the consumption queue will not be maintained. to receive follow-up news to ensure Pingyu consumption. In the scenario of version-order consumption, special attention needs to be paid to the handling of exceptions. If retries fail, the current message will be blocked until the maximum number of retries is exceeded. As a result, subsequent messages cannot be consumed for a long period of time, resulting in queued messages. accumulation.
Technical principles of concurrent consumption:
The interviewer pretended to be calm on the surface, but was actually shocked inside. He slowly said: In addition to sequential consumption, are there any other consumption patterns?
I'm really fucked, isn't it enough? ? ?
In fact, there is. Rocke MQ supports two consumption methods: sequential consumption and concurrent consumption. Concurrent consumption is the default consumption method. The most commonly used method in daily development process, besides sequential consumption, is concurrent consumption. Concurrent consumption is also called out-of-order consumption. The principle is that the same message queue is provided to multiple consumer threads in the Consumer to pull consumption. A consumer thread pool is maintained in Consumer, and multiple consumer threads can be sent to the same message queue to pull messages for consumption. If a consumer thread throws an exception while performing business processing in the listener, the message pulled by the current consumer thread will be retried, without affecting the consumption progress of other consumer threads and consumption queues. Threads that consume successfully submit the consumption progress normally. .
Compared with sequential consumption, concurrent consumption does not involve the process of resource contention and locking, and the speed of consuming messages is much faster than sequential consumption.
Idempotency of messages:
Oh, it seems that it’s really not a problem for you. Then let’s talk about the idempotence of messages.
Good interviewer, we often encounter idempotent problems in business processing. Can the same message be consumed multiple times? If received repeatedly, does multiple consumption conform to business logic?
In RocketMQ, it does not guarantee that messages will not be consumed repeatedly. If the business team is very sensitive to repeated consumption, idempotent processing must be performed at the business level. There are many ways to implement it. Of course, it can also be distributed Lock to achieve:
In all message systems, including rabbitMQ, kafka, rocketMQ, there are three modes: at-most-once (at most once), at-least-once (at least once) and exactly-only-once (exactly once), Distributed messaging is a balance between the three. The first two are feasible and widely used.
at-most-once (at most once): After the message is delivered, regardless of whether the consumption is successful, it will not be delivered again, which may result in the consumption not being consumed. RocketMQ does not use this method.
at-least-once (at least once): After the message is delivered, return ACK (message confirmation mechanism) to the server. If there is no consumption, it will definitely not return ACK. Due to network abnormalities and other circumstances, the server failed to receive the ACK returned by the client. , the server will deliver it again, which will lead to duplicate messages. RocketMQ uses ACK confirmation to ensure that the message is consumed at least once.
exactly-only-once (exactly once): Two conditions must be met. First, in the message sending phase, duplicate messages are not allowed to be sent. Second, in the message consumption phase, duplicate messages are not allowed to be consumed. In a distributed environment, if you want to implement this mode, huge overhead is indispensable. In order to pursue high performance, RocketMQ does not guarantee this feature and cannot avoid message duplication. It is changed to business to ensure idempotence.
Okay, young man, it seems that you are doing well and have a good foundation. It is not early today. You can go back first. Let’s talk about business news tomorrow?
Ahem...what a great interviewer! (Inner OS: Are you still here? Do you have free time every day? I think you have more free time than me memorizing eight-part essays every day)
postscript:
To write this seemingly simple article, I also read some books and checked some information, but it is not easy hahaha. I hope it can help you!
In addition, I have written more than half of RocketMQ and it is almost over. If you have anything you want to read in the future, you can tell me in the comments.
By the way, I want to complain again, the input method that comes with Mac is extremely difficult to use! real!