There are many message queue products on the market, such as RabbitMQ, Kafka, ZeroMQ, and Alibaba’s RocketMQ, etc. There are more than a dozen of these well-known products. Based on my own experience and interests, this article only intends to talk about RabbitMQ. Before that, let's look at the related concepts of message queues.
1. Message queue
1.1. Definition
Message (Message) refers to the data transmitted between applications. Messages can be as simple as just text strings, or more complex and may contain embedded objects.
Message Queuing (Message Queue, MQ) is a communication method between applications. Messages can be returned immediately after sending, and the message system ensures reliable delivery of messages. The message publisher only publishes the message to MQ regardless of who gets it, and the message user only gets the message from MQ regardless of who publishes it. In this way, neither the publisher nor the user needs to know the existence of the other party.
To put it bluntly, MQ is a "forwarder". The producer first delivers the message to a container called "queue", then takes the message from the container, and finally forwards it to the consumer, that's all.
For MQ, whether it is RabbitMQ, Kafka or other message queues, their essence is: send, store, and consume . Let's take this essence as the root and talk about MQ from the shallower to the deeper.
1.2. MQ model
All MQ products have the same process from the model abstraction:
the producer (producer) creates a message, then publishes it to the queue (queue), and finally sends the message to the listening consumer.
1.2.1. The original model
The above picture is the most primitive model of message queue, which contains two keywords: message and queue.
1. Message: It is the data to be transmitted, which can be the simplest text string or a custom complex format (as long as it can be parsed according to the predetermined format).
2. Queue: Everyone should be familiar with it. It is a first-in, first-out data structure. It is a container for storing messages. Messages enter the queue from the end of the queue and exit from the head of the queue. The process of sending messages when entering the queue, and the process of receiving messages when leaving the queue.
1.2.2. Queue model
Strictly speaking, the original model is a queue (Queue), that is, the queue model: the order in which messages are written is read out in that order. However, the queue does not have the operation of "reading". Reading means leaving the queue and "deleting" the message from the head of the queue.
The queue model allows multiple producers to send messages to the same queue. However, if there are multiple consumers, it is actually a competitive relationship, that is, a message can only be received by one of the consumers, and it will be deleted after reading it.
1.2.3. Publish-Subscribe Model
If a piece of message data needs to be distributed to multiple consumers, and each consumer is required to receive the full amount of messages. Obviously, the queue model cannot meet this requirement.
One possible solution is to create a separate queue for each consumer and let the producer send multiple copies. This approach is stupid, and the same data will be copied multiple times, which is also a waste of space.
In order to solve this problem, another message model has evolved: the publish-subscribe model.
In the publish-subscribe model, the container for storing messages becomes a "topic", and subscribers need to "subscribe to a topic" before receiving messages. Eventually, each subscriber can receive the full amount of messages on the same topic.
Carefully compare the similarities and differences between it and the "queue mode": the producer is the publisher, the queue is the topic, and the consumer is the subscriber, there is no essential difference. The only difference is: whether a piece of message data can be consumed multiple times.
1.2.4. Application
The above two models are plain: the difference between unicast and broadcast. Moreover, when there is only one subscriber in the publish-subscribe model, it is the same as the queue model, so it is fully compatible with the queue model in function. Modern mainstream RocketMQ and Kafka are implemented directly based on the publish-subscribe model.
From the above description, we can see that message queue is an asynchronous cooperation mechanism between applications, so when do we need to use MQ?
Taking the common order system as an example, after the order payment is successful, it is necessary to update the order status, update user points, notify the merchant of a new order, update the user portrait in the recommendation system, and so on.
In the early stage of business development, these logics may be put together and executed synchronously. As the business develops and the order volume increases, the performance of system services needs to be improved. At this time, some operations that do not need to take effect immediately can be split out and executed asynchronously, such as updating user points. , Notify merchants, etc.
After the introduction of MQ, order payment now only needs to focus on its most important process: updating the order status. All other unimportant things are handed over to MQ to notify. This is the core problem that MQ solves: system decoupling.
Before the transformation, the order system relied on 3 external systems. After the transformation, it only relied on MQ, and the subsequent business expansion (for example: the marketing system intends to reward coupons for payment users), does not involve the modification of the order system, thus ensuring the stability of the core process performance, reducing maintenance costs.
This modification also brings another benefit: because of the introduction of MQ, the steps of updating user points, notifying merchants, and updating user portraits are all executed asynchronously, which can reduce the overall time-consuming order payment and improve the throughput of the order system. This is another typical application scenario of MQ: asynchronous communication.
The above is the case for business decoupling. Other common scenarios include eventual consistency, broadcasting, peak-shift flow control, and so on.
2.RabbitMQ
2.1. Introduction
AMQP, or Advanced Message Queuing Protocol, is an open standard for application layer protocols designed for message-oriented middleware. Message middleware is mainly used for decoupling between components, and the sender of the message does not need to know the existence of the message user, and vice versa.
The main features of AMQP are message-oriented, queue, routing (including point-to-point and publish/subscribe), reliability, and security.
There are some differences between the routing process of messages in AMQP and the JMS familiar to Java developers. The roles of Exchange and Binding are added to AMQP. The producer publishes the message to the Exchange, the message finally reaches the queue and is received by the consumer, and Binding decides which queue the exchange's message should be sent to.
RabbitMQ is an open source AMQP implementation. The server side is written in Erlang language and supports multiple clients, such as: Python, Ruby, .NET, Java, JMS, C, PHP, ActionScript, XMPP, STOMP, etc., and supports AJAX. It is used to store and forward messages in a distributed system, and performs well in terms of ease of use, scalability, and high availability.
2.2. Model
The model structure of RabbitMQ is mainly composed of the following parts:
- Message
message, the message is anonymous, it consists of a message header and a message body. The message body is opaque, while the message header consists of a series of optional attributes, including routing-key (routing key), priority (priority relative to other messages), delivery-mode (indicating that the message may require persistent storage), etc. - Producer
The producer of the message is also a client application that publishes the message to the exchange. - Exchange
exchange, used to receive messages sent by the producer and route these messages to the queue in the server. - Binding
binding, used for the association between the message queue and the exchange. A binding is a routing rule that connects an exchange and a message queue based on a routing key, so an exchange can be understood as a routing table composed of bindings. - Queue
message queue, used to save messages until sent to consumers. It is the container of the message and the destination of the message. A message can be put on one or more queues. The message is always in the queue, waiting for the consumer to connect to the queue to take it away. - Virtual Host
, which represents a batch of switches, message queues and related objects. Virtual hosts are separate domains of servers that share the same authentication and encryption environment. Each vhost is essentially a mini version of RabbitMQ server, with its own queue, switch, binding and permission mechanism. The vhost is the basis of the AMQP concept and must be specified when connecting. The default vhost of RabbitMQ is /. - Broker
represents the message queue server entity. - Virtual Host
, which represents a batch of switches, message queues and related objects. When multiple different users use the services provided by the same RabbitMQ server, multiple vhosts can be divided, and each user creates and has its own queue, switch, binding and permission mechanism in its own vhost. - Broker
represents the message queue server entity, whose role is to maintain a route from Producer to Consumer to ensure that data can be transmitted in a specified manner. - Connection
network connection, such as a TCP connection, both Producer and Consumer are connected to RabbitMQ Server through TCP. - Channel
channel, an independent bidirectional data flow channel in a multiplexed connection. The channel is a virtual connection built on the real TCP connection. AMQP commands are sent through the channel. Whether it is publishing a message, subscribing to a queue or receiving a message, these actions are all completed through the channel. Because it is very expensive for the operating system to establish and destroy TCP, the concept of channel is introduced to reuse a TCP connection. - Consumer
The consumer of the message represents a client application program that obtains messages from the message queue.
2.2.1. Producer sending message process :
- Producer and Broker establish a TCP connection.
- Producer and Broker establish a channel.
- The Producer sends the message to the Broker through the channel, and the Exchange forwards the message.
- Exchange forwards the message to the specified Queue (queue)
2.2.2. Consumers receive message process:
- Consumer and Broker establish a TCP connection
- Consumer and Broker establish a channel
- Consumer listens to the specified Queue (queue)
- When a message arrives on the Queue, the Broker pushes the message to the Consumer by default.
- Consumer receives the message.
- ACK reply (Consumer sends receipt).
Note: RabbitMQ removes the message from the Queue only after receiving the message acknowledgment; if RabbitMQ does not receive the acknowledgment and detects that the consumer's RabbitMQ connection is disconnected, RabbitMQ will send the message to other consumers (if multiple consumers exist) for processing. There is no timeout concept here. No matter how long a consumer processes a message, it will not cause the message to be sent to other consumers unless its RabbitMQ connection is disconnected.
There will be another problem here. If our developers forget to send a receipt to RabbitMQ after processing the business logic, this will lead to serious bugs - more and more messages will be accumulated in the Queue; after the consumer restarts, it will Repeated consumption of these messages and repeated execution of business logic...
2.3. Exchange type
When Exchange distributes messages, there are different distribution strategies according to different types. Currently, there are four types: direct, fanout, topic, and headers.
2.3.1.direct
direct switch: If the routing key in the message is consistent with the binding key in Binding, the switch will send the message to the corresponding queue.
The routing key exactly matches the queue name. If a queue is bound to an exchange and requires the routing key to be "dog", only messages whose routing key is marked as "dog" will be forwarded, neither "dog.puppy" nor "" dog. guard" and so on. It is an exact match, unicast mode.
2.3.2.fanout
Fanout exchange: Every message sent to a fanout exchange will be assigned to all bound queues.
The fanout exchange does not handle routing keys, but simply binds queues to the exchange, and every message sent to the exchange will be forwarded to all queues bound to the exchange. Much like subnet broadcasting, every host on the subnet gets a copy of the message. The fanout type forwarding messages is the fastest.
2.3.3.topic
The exchange of the topic type has been extended on the matching rules. It is similar to the exchange of the direct type. It also routes the message to the Queue whose binding key matches the routing key. However, the matching rules here are somewhat different. It agrees:
- The routing key is a string separated by a period "." (we call each independent string separated by a period "." a word), such as "usa.news", "usa.weather" wait;
- The binding key is the same as the routing key, which is also a string separated by a period ".";
- There can be two special characters "*" and "#" in the binding key for fuzzy matching, where "*" is used to match one word, and "#" is used to match multiple words (can be zero).
Take the configuration in the above figure as an example, messages with routingKey=”usa.news” will be routed to Q1 and Q2 at the same time, messages with routingKey=”usa.weather” will be routed to Q1 and Q3, routingKey=”quick.usa.weather. news.rabbit" will be discarded because they do not match any bindingKey.
2.3.4.headers
The exchange of headers type does not rely on the matching rules of routing key and binding key to route messages, but matches according to the headers attribute in the content of the sent message.
Specify a set of key-value pairs when binding Queue and Exchange; when a message is sent to Exchange, RabbitMQ will get the headers of the message (also in the form of a key-value pair), and compare whether the key-value pair exactly matches Queue and Exchange. The key-value pair specified when Exchange is bound; if there is an exact match, the message will be routed to the Queue, otherwise it will not be routed to the Queue.
This type of Exchange has never been used (but it should be useful), so I won't introduce it.
2.4. Summary
This article introduces the concepts that I personally think are the most important in RabbitMQ. Making full use of these functions provided by RabbitMQ can handle most of our asynchronous business.
The basic concept of this article may be difficult to understand and digest, and it should be easier to absorb when combined with the actual application code, so the installation and use of RabbitMQ will be written in the next article.