Original address: http://mp.weixin.qq.com/s/Brd-j3IcljcY7BV01r712Q
1. Origin
All architectural designs and new technology introductions that are out of business are hooligans.
Before introducing a technology, the first question that should be answered is what problem does the technology solve.
Just like before the layered architecture of microservices, you should first answer why microservices are introduced and what problems do microservices solve (see " Why Internet Architecture Should Be Microservices? ").
Recently shared several MQ related articles:
" How MQ Implements Delayed Messages "
" How MQ realizes the message must be reached "
" How MQ achieves idempotency "
Many netizens have asked when to use MQ and what scenarios are MQ suitable for, so this article is written.
2. What is MQ?
Message Queue, hereinafter referred to as MQ, is a cross-process communication mechanism used to transmit messages upstream and downstream.
In the Internet architecture, MQ is a very common upstream and downstream "logical decoupling + physical decoupling" message communication service.
After using MQ, the upstream of message sending only needs to rely on MQ, and does not need to rely on other services both logically and physically.
3. When not to use the message bus
Since MQ is a decoupling tool in the layered architecture of the Internet, wouldn't it be good to use MQ for all communications? This is a serious misunderstanding. The relationship between calling and being called cannot be replaced by MQ.
The shortcomings of MQ are:
1) The system is more complex, with an additional MQ component
2) The message transmission path is longer, and the delay will increase
3) Message reliability and repeatability are contradictory, and it is difficult to ensure that messages are not lost or heavy at the same time
4) The upstream cannot know the execution result of the downstream, which is fatal
For example, in a user login scenario, the login page calls the passport service, and the execution result of the passport service directly affects the login result. The "login page" and "passport service" here must use the calling relationship instead of MQ communication.
In any case, keep this conclusion in mind: in a business scenario where the caller depends on the execution result in real time, please use the call, not MQ.
Fourth, when to use MQ
[Typical Scenario 1: Data-Driven Task Dependency]
What is task dependency? For example, Internet companies often perform some data statistics tasks in the early morning, and there are certain dependencies between these tasks, such as:
1) task3 needs to use the output of task2 as input
2) task2 needs to use the output of task1 as input
In this case, there is a task dependency between task1, task2, and task3. Task1 must be executed first, then task2, and then task3.
For such requirements, a common implementation is to use cron to manually schedule the execution:
1) task1, executed at 0:00, the experience execution time is 50 minutes
2) task2, executed at 1:00 (reserve a 10-minute buffer for task1), and the experience execution time is also 50 minutes
3) task3, executed at 2:00 (reserve a 10-minute buffer for task2)
The downside of this approach is:
1) If the execution time of a task exceeds the reserved buffer time, an incorrect result will be obtained, because the post-task does not know whether the pre-task is successfully executed. At this time, the task needs to be manually rerun, and it may be necessary to adjust the schedule. class schedule
2) The execution time of the total task is very long, and a lot of buffers are always reserved. If the pre-task is completed in advance, the post-task will not start in advance
3) If a task is depended on by multiple tasks, this task will be called the critical path, and it is difficult for the schedule to reflect the dependencies, and it is easy to make mistakes
4) If there is one task's execution time to be adjusted, there will be multiple tasks' execution time to be adjusted
In any case, using the "cron schedule" method, each task is coupled, and whoever has used it knows who hurts (please leave a comment if you use this method)
The optimization solution is to use MQ decoupling:
1) Task1 starts on time and sends a "task1 done" message after the end
2) task2 subscribes to the "task1 done" message, starts the execution as soon as it receives the message, and sends a "task2 done" message after the end
3) The same is true for task3
The advantages of using MQ are:
1) There is no need to reserve buffer. After the upstream task is executed, the downstream task will always be executed at the first time
2) Depends on multiple tasks, and is easily handled by multiple tasks, just need to subscribe to related messages
3) There are changes in task execution time, and downstream tasks do not need to adjust the execution time
It should be noted that MQ is only used to transmit the message that the execution of the upstream task is completed, and is not used to transmit the real input and output data.
[Typical scenario 2: The upstream does not care about the execution result]
When the upstream needs to pay attention to the execution result, "call" is used. When the upstream does not pay attention to the execution result, MQ can be used.
For example, many downstream users of 58.com need to pay attention to the event of "user posting". For example, after recruiting users to post posts, the recruitment business will reward 58 beans. After real estate users post posts, real estate business will send 2 stickers to the top, and second-hand users will post them. After the post, the second-hand business wants to modify the user statistics.
For such requirements, the common implementation is to use the call relationship:
After the post publishing service is executed, the downstream recruitment business, real estate business, and second-hand business are called to complete the notification of the news, but in fact, the post publishing service does not pay attention to whether the notification is executed normally and correctly.
The downside of this approach is:
1) The execution time of the post publishing process has increased
2) Downstream services are down, which may affect the post publishing service, and the upstream and downstream logical + physical dependencies are serious
3) Whenever a downstream that needs to know the information of "post publishing successfully" is added, it is the post publishing service that modifies the code, which is the most disgusting, and belongs to the typical dependency inversion in architecture design. Please comment on this method)
The optimization solution is to use MQ decoupling:
1) After the post is published successfully, send a message to MQ
2) Which downstream pays attention to the message "Post published successfully" and actively subscribes to MQ
The advantages of using MQ are:
1) Short upstream execution time
2) The upstream and downstream logic + physical decoupling, except for the physical connection with MQ, the modules are not dependent on each other
3) Add a downstream message follower, upstream does not need to modify any code
Typical scenario 3: The upstream pays attention to the execution result, but the execution time is very long
Sometimes the upstream needs to pay attention to the execution result, but the execution result takes a long time (typically calling offline processing, or calling across the public network), and the callback gateway + MQ is often used for decoupling.
For example, WeChat payment, calling the WeChat interface across the public network, the execution time will be relatively long, but the caller is very concerned about the execution result. How to play at this time?
Generally, the "callback gateway + MQ" scheme is used to decouple:
1) The caller directly calls the WeChat interface across the public network
2) WeChat returns a successful call, which does not mean the return is successful at this time
3) After the execution of WeChat is completed, the unified gateway is called back
4) The gateway will notify MQ of the return result
5) The requester receives the result notification
It should be noted here that the callback gateway should not call the upstream to notify the result. If this is the case, every time a caller is added, the callback gateway needs to modify the code, and it will still rely on the reverse. Use the callback gateway + MQ scheme , any new calls to WeChat payment do not need to modify the code.
V. Summary
MQ is a common decoupling tool in Internet architecture.
When not to use MQ?
Upstream pays attention to execution results in real time
When to use MQ?
1) Data-driven task dependencies
2) The upstream does not care about the results of multiple downstream executions
3) Asynchronous return execution time is long