Today we will talk about the push-pull mode of message queues, which is also an interview hotspot. For example, if you wrote RocketMQ in your resume, you will basically ask you whether RocketMQ uses push mode or pull mode? Is it a pull mode? Isn’t there a PushConsumer?
Today we will talk about the push-pull model, and then take a look at how RocketMQ and Kafka do it.
Push-pull mode
First of all, clarify which step of the message queue is in the push-pull mode. Generally speaking, when we talk about the push-pull mode, we refer to the interaction between Comsumer and Broker.
The default is that between Producer and Broker is the push method, that is, Producer pushes messages to Broker instead of Broker actively pulling messages.
Imagine that if Broker is required to pull messages, then Producer must save messages locally in the form of logs to wait for Broker to pull them. If there are many producers, the reliability of messages depends not only on Broker itself, but also Rely on hundreds of Producers.
Broker can also rely on mechanisms such as multiple copies to ensure the reliable storage of messages. The reliability of hundreds of producers is a bit difficult, so the default Producer pushes messages to Broker.
So in some cases it is better to be distributed, but in some cases centralized management is better.
Push mode
Push mode refers to the message being pushed from the Broker to the Consumer, that is, the Consumer passively receives the message, and the Broker takes the lead in sending the message.
Let's think about the benefits of push mode?
The real-time message is high, and the Broker can push it to the Consumer immediately after receiving the message.
It's easier for consumers to use, simple, just wait, and any news will be pushed over.
What are the disadvantages of push mode?
The push rate is difficult to adapt to the consumption rate. The goal of the push model is to push messages at the fastest speed. When the rate of the producer sending messages to the Broker is greater than the rate at which consumers consume messages, the consumer may become " "Broken up", because there is no consumption at all. When the push rate is too fast, like a DDos attack, consumers are stupid.
And the consumption rate of different consumers is not the same. As a Broker, it is difficult to balance the push rate of each consumer. If you want to achieve an adaptive push rate, you need to tell the Broker when the consumer is pushing, I can’t do you Push slowly, and then Broker needs to maintain the status of each consumer to change the push rate.
This actually increases the complexity of Broker itself.
Therefore, the push mode is difficult to control the push rate according to the state of the consumer, and it is suitable for situations where the volume of messages is small and the consumption power is high and real-time is required.
Pull mode
The pull mode refers to that the Consumer actively requests the Broker to pull messages, that is, the Broker passively sends messages to the Consumer.
Let's think about the benefits of pull mode?
The initiative in the pull mode lies with consumers, who can initiate pull requests based on their own conditions. Assuming that the current consumer feels that he can't afford it, he can stop the pull according to a certain strategy, or pull it at intervals.
Broker is relatively easy in pull mode. It only saves the messages sent by the producer. As for consumption, it is naturally initiated by the consumer. A request is given to it. The message is taken from where to get the message. Tell it that it is a tool person with no emotions, and it does not matter if consumers do not come to pick it up.
The pull mode can be more suitable for sending messages in batches. Based on the push mode, one message can be pushed, or some messages can be cached and then pushed, but when pushing, it is not known whether consumers can handle so many messages at once. The pull mode is more reasonable. It can refer to the information requested by the consumer to determine how many messages to cache and send in batches.
What are the disadvantages of pull mode?
Message delay, after all, is the consumer to pull the message, but how does the consumer know that the message has arrived? So it can only pull continuously, but it cannot request it very frequently. If it is too frequent, it will become a consumer attacking the Broker. Therefore, you need to reduce the frequency of requests. For example, if you request once every 2 seconds, you may be delayed by 2 seconds when you look at the message.
Message busy request. Busy request means that, for example, the message is received several hours later, then the consumer's request is invalid within a few hours, and it is doing useless work.
Is that pushing or pulling
We can see that the push mode and the pull mode have their own advantages and disadvantages. How to choose?
Both RocketMQ and Kafka have chosen the pull model. Of course, the industry also has push model-based message queues such as ActiveMQ.
I personally think that the pull model is more appropriate, because the current message queue has the need for persistent messages, that is to say, it has a storage function itself, and its mission is to receive messages and save the messages so that consumers can consume them. can.
Consumers are diverse. As a Broker, you should not have the tendency to rely on consumers. I have saved the news for you, and you can take it if you want.
Although in general, Broker will not become a bottleneck, because the consumer has slower business consumption, but Broker is a central point after all, and it is as light as possible.
So RocketMQ and Kafka both chose the pull mode, are they not afraid of the shortcomings of the pull mode? Scared, so they operate a wave, alleviating the shortcomings of the pull mode.
Long polling
RocketMQ and Kafka both use "long polling" to implement the pull model, let's take a look at how they operate.
For the sake of simplicity, below, I will uniformly describe the news that does not meet the number and total size of this pull as there is no news, anyway, the conditions are not met.
Long polling in RocketMQ
PushConsumer in RocketMQ is actually a method of wearing a pull mode, but it looks like a push mode.
Because RocketMQ secretly helped us go to the Broker to request data behind the scenes.
There will be a RebalanceService thread in the background. This thread will perform load balancing based on the number of topic queues and the number of consumers in the current consumer group. The pullRequest generated by each queue is put into the blocking queue pullRequestQueue. Then there is another PullMessageService thread that continuously obtains pullRequest from the blocking queue pullRequestQueue, and then requests the broker through the network, so that it can pull messages in quasi-real time.
I won't cut off this part of the code, it's just such a thing, I will show it with a picture later.
Then the processRequest method in Broker's PullMessageProcessor is used to process the pull message request. If there is a message, it will be returned directly. What if there is no message? Let's take a look at the code.
Let's look at what the suspendPullRequest method does.
The PullRequestHoldService thread will fetch the PullRequest request from the pullRequestTable every 5 seconds, and then check whether the offset of the message request to be pulled is less than the maximum offset of the current consumption queue. If the condition is true, it means that there is a new message, and notifyMessageArriving will be called. Finally, the executeRequestWhenWakeup() method of PullMessageProcessor is called to retry to process the request for this message, that is, to do it again. The default long polling time is 30 seconds.
Simply put, it is time to check the message every 5 seconds, and if it is, call processRequest to process it again. Doesn't it seem real time? 5 seconds?
Don’t worry, there is also a ReputMessageService thread. This thread is used to continuously parse data from commitLog and distribute requests, constructing two types of data, ConsumeQueue and IndexFile, and there will also be wake-up requests to make up for every 5s. Slow delay
I will not intercept the code, that is, the message is written and pullRequestHoldService#notifyMessageArriving will be called.
Finally, I will draw a picture to describe the whole process.
Long polling in Kafka
Like Kafka, there are parameters in the pull request, which can make consumer requests block waiting in the "long polling".
Simply put, the consumer goes to the Broker to pull the message and defines a timeout period, which means that the consumer requests the message and returns the message immediately if there is any. If not, the consumer waits until the timeout, and then initiates the pull message request again .
And the Broker must also cooperate. If the consumer requests it, a message must be returned immediately. If there is no message, a delayed operation is established, and the condition is met before returning.
Let's take a brief look at the source code. In order to highlight the key points, I will cut some code.
Let's look at the consumer code first.
The poll interface above must be familiar to everyone. In fact, you know directly from the comments that it is indeed waiting for the arrival of data or timeout. Let's look down briefly.
Let's take a look at what the final client.poll calls.
The last call is the selector wrapped by Kafka, and finally select(timeout) of Java nio is called.
Now that the consumer code is clear, let's take a look at how Broker does it.
The entry for Broker to process all requests was actually introduced in the previous article, just under the handle method of the KafkaApis.scala file, this time the protagonist is handleFetchRequest.
This method comes in, I intercept the most important part.
The following picture is the internal implementation of the fetchMessages method. The comments given by the source code are already very clear. You can zoom in and see.
The name of this purgatory is very interesting. Simply put, it is to use the time wheel mentioned in my previous article to perform timing tasks. For example, it is delayedFetchPurgatoryspecifically used to handle delayed pull operations.
Let’s first think briefly about what methods need to be implemented for this delayed operation. First, the delayed operation constructed needs to have a check mechanism to check whether the message has arrived, and then there must be a method that should be executed after the message arrives, and it needs to be executed. What to do after finishing, of course, there must be a method to do after timeout.
These methods actually correspond to DelayedFetch in the code. This class inherits DelayedOperation and has:
isCompleted method to check whether the condition is satisfied
tryComplete method to be executed after the condition is met
The method called after onComplete is executed
The method that needs to be executed after onExpiration expires
Judging whether it is overdue is driven by the time wheel, but you can’t wait to see the news when it’s over, right?
The mechanism of Kafka and RocketMQ here is the same. It will also remind these delayed request messages when the message is written. I will not post the specific code. You can see two more methods in the ReplicaManager#appendRecords method.
Although the code is not posted, the picture still needs to be drawn.
Summary
It can be seen that both RocketMQ and Kafka adopt the "long polling" mechanism. The specific method is to wait for messages through consumers. When there is a message, the broker will directly return the message. If there is no message, it will adopt a delay processing strategy, and In order to ensure the timeliness of the message, when a new message arrives in the corresponding queue or partition, it will remind the message to come and return the message in time.
In a word, the consumer and Broker cooperate with each other to hold when the pull message request does not meet the conditions, avoiding multiple frequent pull actions, and reminding to return as soon as the message arrives.
At last
Generally speaking, the push-pull mode has its own advantages and disadvantages, and I personally think that the pull-down mode is more suitable for message queues in general.
There is no way, but the technique can be achieved; if there is no way, it ends with the technique
Welcome everyone to follow the Java Way public account
Good article, I am reading ❤️