In every age, we will never treat people who can learn
Hello everyone, I am yes.
Today we will talk about the push-pull mode of message queues, which is also an interview hotspot. For example, if you wrote RocketMQ in your resume, you will basically ask you whether RocketMQ uses a push mode or a pull mode? Is it pull mode? Isn’t there a PushConsumer?
Today we will talk about the push-pull model, and let's take a look at how RocketMQ and Kafka do it.
Push-pull mode
First of all, clarify which step of the message queue is being discussed in the push-pull mode . Generally speaking, when we talk about the push-pull mode, we refer to the interaction between Comsumer and Broker .
The default is that between the Producer and the Broker is the push method, that is, the Producer pushes the message to the Broker, rather than the Broker actively pulling the message.
Imagine that if Broker is required to pull messages, then Producer must save messages locally in the form of logs to wait for Broker to pull them. If there are many producers, the reliability of messages depends not only on Broker itself, but also Rely on hundreds of Producers.
Broker can also rely on mechanisms such as multiple copies to ensure the reliable storage of messages. The reliability of hundreds of producers is a bit difficult, so the default Producer is to push messages to Broker.
Therefore, in some cases, it is better to be distributed, but in some cases centralized management is better.
Push mode
Push mode refers to the message being pushed from the Broker to the Consumer, that is, the Consumer passively receives the message, and the Broker takes the lead in sending the message.
Let's think about the benefits of push mode?
The real-time message is high , and the Broker can push it to the Consumer immediately after receiving the message.
It's easier for consumers to use, simple , just wait, and any news will be pushed over.
What are the disadvantages of push mode?
The push rate is difficult to adapt to the consumption rate . The goal of the push model is to push messages at the fastest speed. When the rate of the producer sending messages to the Broker is greater than the rate at which consumers consume messages, the consumer may become " "Broken up", because there is no consumption at all. When the push rate is too fast, like a DDos attack, consumers are stupid.
And the consumption rate of different consumers is not the same. As a Broker, it is difficult to balance the push rate of each consumer. If you want to achieve an adaptive push rate, you need to tell the Broker when the consumer is pushing, I can’t do you Push slowly, and then Broker needs to maintain the state of each consumer to change the push rate.
This actually increases the complexity of Broker itself.
Therefore, the push mode is difficult to control the push rate according to the state of the consumer, and it is suitable for situations where the volume of messages is small and the consumption power is high and real-time is required.
Pull mode
The pull mode refers to that the Consumer actively requests the Broker to pull messages, that is, the Broker passively sends messages to the Consumer.
Let's think about the benefits of pull mode?
The initiative in the pull mode lies with consumers, who can initiate requests to pull messages according to their own circumstances . Assuming that the current consumer feels that he can't afford it, he can stop the pull according to a certain strategy, or pull it at intervals.
In the pull mode, Broker is relatively easy . It only stores the messages sent by the producer. As for consumption, it is naturally initiated by the consumer. A request is given to it. The message is taken from where you can get the message. Tell it that it is a tool person with no emotions, and it does not matter if consumers do not come to get it.
The pull mode can be more suitable for batch sending of messages . Based on the push mode, one message can be pushed, or some messages can be cached and then pushed, but when pushing, it is not known whether consumers can handle so many messages at once. The pull mode is more reasonable. It can refer to the information requested by consumers to determine how many messages to cache and send in batches.
What are the disadvantages of pull mode?
Message delay , after all, is the consumer to pull the message, but how does the consumer know that the message has arrived? So it can only pull continuously, but it cannot request it very frequently. If it is too frequent, it will become a consumer attacking Broker. Therefore, you need to reduce the frequency of requests. For example, if you request once every 2 seconds, you may see the message delayed by 2 seconds.
Message busy request . Busy request means that, for example, the message is received several hours later, then the consumer's request is invalid within a few hours, and it is doing useless work.
Is that pushing or pulling
We can see that the push mode and the pull mode have their own advantages and disadvantages. How to choose?
Both RocketMQ and Kafka have chosen the pull model. Of course, the industry also has push model-based message queues such as ActiveMQ.
I personally think that the pull model is more appropriate, because the current message queue has the need for persistent messages, which means that it has a storage function itself. Its mission is to accept messages and save the messages so that consumers can consume messages. can.
There are various consumers. As a Broker, you should not have a tendency to rely on consumers. I have saved the news for you, and you can take it if you want.
Although in general, Broker will not become a bottleneck, because the consumer has slower business consumption, but Broker is a central point after all, as light as possible.
So RocketMQ and Kafka both chose the pull mode, are they not afraid of the shortcomings of the pull mode? Scared, so they operate a wave, alleviating the shortcomings of the pull mode.
Long polling
RocketMQ and Kafka both use "long polling" to implement the pull mode, let's take a look at how they operate.
For the sake of simplicity, below, I will uniformly describe the news that does not meet the number and total size of this pull as there is no news, anyway, the conditions are not met.
Long polling in RocketMQ
PushConsumer in RocketMQ is actually a method of wearing a pull mode, but it looks like a push mode .
Because RocketMQ secretly helped us go to Broker to request data behind the scenes.
There will be a RebalanceService thread in the background. This thread will perform load balancing based on the number of topic queues and the number of consumers in the current consumer group. The pullRequest generated by each queue is put into the blocking queue pullRequestQueue. Then there is another PullMessageService thread that continuously obtains pullRequest from the blocking queue pullRequestQueue, and then requests the broker through the network, so that it can pull messages in quasi-real time.
I won't cut off this part of the code, it's just such a thing, and I will show it with a picture later.
Then the processRequest method in Broker's PullMessageProcessor is used to process the pull message request. If there is a message, it will be returned directly. What if there is no message? Let's take a look at the code.
Let's look at what the suspendPullRequest method does.
The PullRequestHoldService thread will fetch the PullRequest request from the pullRequestTable every 5 seconds, and then check whether the offset of the message request to be pulled is less than the maximum offset of the current consumption queue. If the condition is true, it means that there is a new message, and notifyMessageArriving will be called. Finally, the executeRequestWhenWakeup() method of PullMessageProcessor is called to retry to process the request for this message, that is, to do it again. The default long polling time is 30 seconds.
Simply put, it is time to check the message every 5 seconds, and if it is, call processRequest to process it again. Doesn't it seem real time? 5 seconds?
Don’t worry, there is also a ReputMessageService thread. This thread is used to continuously parse data from commitLog and distribute requests, constructing two types of data, ConsumeQueue and IndexFile, and there will also be wake-up requests to make up for every 5s. Slow delay
I will not intercept the code, that is, the message is written and pullRequestHoldService#notifyMessageArriving will be called.
Finally, I will draw a diagram to describe the entire process.
Long polling in Kafka
Like Kafka, there are parameters in pull requests, which can make consumer requests block waiting in "long polling".
Simply put, the consumer goes to the Broker to pull the message and defines a timeout period, which means that the consumer requests the message, and returns the message immediately if there is any. If not, the consumer waits until the timeout, and then initiates the message pull request again .
And the Broker must also cooperate. If the consumer requests it, a message must be returned immediately. If there is no message, a delayed operation is established, and the condition is met before returning.
Let's take a brief look at the source code. In order to highlight the key points, I will cut some code.
Let's first look at the consumer code.
The poll interface above must be familiar to everyone. In fact, you know directly from the comments that it is indeed waiting for the arrival of data or timeout. Let's look down briefly.
Let's look at what the final client.poll calls.
The last call is the selector wrapped by Kafka, and finally select(timeout) of Java nio is called .
Now that the consumer code is clear, let's take a look at how Broker does it .
The entry for Broker to process all requests was actually introduced in the previous article, just under the handle method of the KafkaApis.scala file, this time the protagonist is handleFetchRequest.
This method comes in, I intercept the most important part.
The following picture is the internal implementation of the fetchMessages method. The comments given by the source code are already very clear. You can zoom in and see.
The name of this purgatory is very interesting. Simply put, it is to use the time wheel mentioned in my previous article to perform timing tasks. For example, it is delayedFetchPurgatoryspecifically used to handle delayed pull operations.
Let's briefly think about what methods need to be implemented for this delay operation. First, the delay operation constructed needs to have a check mechanism to check whether the message has arrived, and then there must be a method to be executed after the message arrives, and it needs to be executed. What to do after finishing, of course, there must be a method to do after timeout.
These methods actually correspond to DelayedFetch in the code. This class inherits DelayedOperation and has:
- isCompleted method to check whether the condition is satisfied
- tryComplete method to be executed after the condition is met
- The method called after onComplete is executed
- The method that needs to be executed after onExpiration expires
Judging whether it has expired is driven by the time wheel, but you can't wait until it expires to see the news. Has it arrived?
The mechanism of Kafka and RocketMQ here is the same. It will also remind these delayed request messages when the message is written. I will not post the specific code. You can see two more methods in the ReplicaManager#appendRecords method.
Although the code is not posted, the picture still needs to be drawn.
Summary
It can be seen that RocketMQ and Kafka both use the "long polling" mechanism. The specific method is to wait for messages through consumers. When there is a message, the broker will directly return the message. If there is no message, it will adopt a delay processing strategy, and In order to ensure the timeliness of the message, when a new message arrives in the corresponding queue or partition, it will remind the message to come and return the message in time.
In a word, the consumer and Broker cooperate with each other to hold when the pull message request does not meet the conditions, avoiding multiple frequent pull actions, and reminding to return as soon as the message arrives.
At last
Generally speaking, the push-pull mode has its own advantages and disadvantages, and I personally think that the pull-down mode is more suitable for message queues in general.
After reading this article, I believe the interviewer asked you to push or pull? Suggest to give him a crooked smile.
I am yes, from a little bit to a little bit, see you in the next article .