What is a message queue
Maybe you've heard the term "message queue" a lot, but haven't really used it. It plays a very important role in system architecture. So what exactly is a "message queue"? Today I will talk to you about the "message queue" technology.
Literally analyzed, first of all it has a queue, and secondly, messages are stored in this queue. What the hell is that queue? Don't think too complicated, the queue is a "container" that can store data, in which you can store numbers and strings.
The "message queue" is a middleware service that specifically stores message data. This message is actually a data object one by one. For example, it can be a file path or an order information.
Why use message queues
Before giving the answer, let me give a small example.
There is a process in a small workshop factory, which needs to screw the cap on the bottle and then stick the label. Worker A is good at screwing caps, and worker B is good at labeling, so this process is completed by A and B.
At the beginning, Little A will give it to Little B after twisting the cap, and then put the trademark on. Only after Little B completed the last labeling action, Little A can hand over this bottle to Little B, and then continue to unscrew the next one.
This kind of cooperation requires that the working rhythms of Little A and Little B are completely consistent so as not to waste time.
The boss, Rhubarb, is very smart. He made a bottle rack that can rotate automatically. It is in the shape of a disc and can hold up to 10 bottles at the same time. Little A and Little B are sitting on opposite sides of the disc. He asked Little A to finish screwing the cap Finally, put the bottle on the shelf, and then little B will take the bottle that little A has twisted from the shelf.
In this way, you don’t have to worry about the problem that little A is faster than little B. On the other hand, if little B is faster than little A, then you need to let little B rest for a while, let little A twist a few more bottles first, and little B start working again.
After such an operation, Rhubarb found that the efficiency did not improve, but the rest time for Little A or Little B was extended. At this time, the boss's solution is, if little A is faster than little B, then when little A finishes screwing all 10 bottle caps, he will help little B put labels on it first, otherwise, little B will help little A twist the caps.
Combined with this case, the message queue is actually an automatic rotating bottle rack designed by Rhubarb. We can put the message in the message queue (put it on the shelf after screwing the lid), and when the message is needed, take it out of the message queue for processing (Little B took the bottle with the screwed cap to put a label on it).
The message queue is an important component in a distributed system. The main purpose of using the message queue is to improve system performance, peak shaving, and reduce system coupling through asynchronous processing.
Scenario 1: Decoupling
The so-called decoupling is to split multiple actions of a transaction. For example, in the above example, the original process includes screwing the bottle cap (A) and labeling (B). At the beginning, the two actions are coupled together. , coupled together means that A is slow or B is slow, which will affect the efficiency of the entire process. So now split A and B, A can only focus on twisting the bottle cap, and put it on the shelf (in the message queue) after twisting, while B only needs to get it from the shelf (in the message queue). As long as the shelf capacity is large enough, neither A nor B will be affected by each other.
Going back to the message queue, A and B are two modules respectively, the A module has a message producer, and the B module has a message consumer. The two modules of A and B are decoupled by this message queue.
Scenario 2: Asynchronous
Asynchrony is to separate multiple operations that were originally connected in series so that they no longer wait for each other. For example, in the above example, A does not need to wait for whether B is free after screwing the bottle cap, he just needs to screw the bottle cap, and then puts the screwed bottle on the shelf, and B just takes it from the shelf and twists it. A bottle with a good bottle cap is used to affix the trademark, A and B are asynchronous, and the asynchronous realization needs to use this message queue.
To give another practical example, an audit system needs to upload pictures, audio, and video on the mobile client to the server for auditing. One step in this process is to transcode the video and audio. The purpose is to make multimedia files smaller so that browsers can play them directly. The operation of uploading audio and video is very frequent and the volume is large. If each uploaded file is directly transcoded, it will not only consume server resources, but also directly reduce the server processing speed. So, what we do is, make it asynchronous. Uploading and transcoding are separated, not serialized, and even a certain server can handle the transcoding operation independently. Then this implementation needs to use the message queue to put the transcoding task and key transcoding parameters into the message queue, and the transcoding server then fetches the information from the message queue to transcode. Of course, there is a prerequisite for designing asynchronously, that is, the transcoding operation is not very urgent, and there is no need to transcode immediately after uploading a file.
Scenario 3: peak clipping
Still the above example, one day the boss Rhubarb received a lot of orders, so he called on the whole family to come to work, but everyone was not very good at labeling, so they all went to twist the bottle caps, so they twisted them all at once in the morning I bought 1w bottle caps and put them on the shelf (assuming the boss built a very large shelf). If the normal process is followed, it will definitely take a lot of people who stick labels to quickly digest the 1w bottles with screwed caps, but The boss doesn't have that many employees. In the end, the workers who put up the labels can only work overtime for a while to slowly digest these bottles.
In the real production environment, this very large shelf is the message queue service, which can deal with emergencies, and can temporarily put the sudden and sharp increase in business needs in the message queue, and then consume (process) slowly, which is Clipping. The 12306 ticket grabbing is such a scene.
in conclusion
Message queue is an indispensable component in a large-scale distributed architecture. Its existence allows our system to support greater concurrency, improves system response speed, and makes our system more stable and reliable. In short, the advantages of introducing message queue components outweigh the disadvantages.
So what are the common message queue middleware? Please pay attention to my official account, and I will introduce them one by one later.