Have you ever encountered the need to synchronize some data between two (multiple) systems through scheduled tasks? Are you troubled and struggling with the problem of calling and communicating with each other between different processes in a heterogeneous system? If so, congratulations, the messaging service makes it easy for you to resolve these issues.
The message service is good at solving the problem of data exchange (message notification/communication) between multiple systems and heterogeneous systems, and you can also use it for mutual invocation (RPC) of services between systems. RabbitMQ, which will be introduced in this article, is one of the most mainstream message middleware at present.
Introduction to RabbitMQ
AMQP, the Advanced Message Queuing Protocol, is an open standard for application layer protocols, designed for message-oriented middleware. Message middleware is mainly used for decoupling between components, the sender of the message does not need to know the existence of the message consumer, and vice versa.
The main features of AMQP are message-oriented, queuing, routing (including point-to-point and publish/subscribe), reliability, and security.
RabbitMQ is an open source AMQP implementation, the server side is written in Erlang language, supports a variety of clients, such as: Python, Ruby, .NET, Java, JMS, C, PHP, ActionScript, XMPP, STOMP, etc., supports AJAX. It is used to store and forward messages in distributed systems, and it performs well in terms of ease of use, scalability, and high availability.
The following will focus on some basic concepts in RabbitMQ. Understanding these concepts is the basis for using RabbitMQ well.
ConnectionFactory、Connection、Channel
ConnectionFactory, Connection, and Channel are the most basic objects in the API provided by RabbitMQ. Connection is the socket link of RabbitMQ, which encapsulates some logic related to the socket protocol. ConnectionFactory is the manufacturing factory of Connection.
Channel is the most important interface we deal with RabbitMQ. Most of our business operations are completed in the Channel interface, including defining Queue, defining Exchange, binding Queue and Exchange, publishing messages, etc.
Queue
Queue (queue) is an internal object of RabbitMQ, used to store messages, represented by the following figure.
The messages in RabbitMQ can only be stored in the Queue. The producer (P in the figure below) produces the message and finally delivers it to the Queue. The consumer (C in the figure below) can get the message from the Queue and consume it. ( The concept of switch is omitted below )
Multiple consumers can subscribe to the same Queue. At this time, the messages in the Queue will be evenly distributed to multiple consumers for processing, instead of each consumer receiving all the messages and processing them.
Message acknowledgment
In practical applications, it may happen that the consumer receives the message in the Queue, but fails to complete the processing (or has other accidents). In this case, the message may be lost. In order to avoid this situation, we can ask the consumer to send a receipt to RabbitMQ after consuming the message, and RabbitMQ will remove the message from the Queue after receiving the message acknowledgment; if RabbitMQ does not receive the receipt and Detecting that the consumer's RabbitMQ connection is disconnected, RabbitMQ will send the message to other consumers (if there are multiple consumers) for processing. There is no timeout concept here, and a consumer processing a message for a long time will not cause the message to be sent to other consumers unless its RabbitMQ connection is disconnected.
Another problem will arise here. If our developers forget to send a receipt to RabbitMQ after processing the business logic, this will lead to serious bugs - more and more messages will accumulate in the Queue; Consume these messages repeatedly and execute business logic repeatedly...
Message durability
By default, the message queues we create and the messages stored in the queues are non-persistent, that is to say, when RabbitMQ goes down or restarts, the created message queues and messages will not be saved. In this case, to ensure the reliability of message transmission, we can use the persistence mechanism of the message queue provided by RabbitMQ.
Prefetch count
Earlier we mentioned that if multiple consumers subscribe to messages in the same Queue at the same time, the messages in the Queue will be shared among multiple consumers. At this time, if the processing time of each message is different, it may cause some consumers to be busy all the time, while other consumers quickly finish the work at hand and remain idle. We can limit the number of messages that the Queue sends to each consumer each time by setting prefetchCount. For example, if we set prefetchCount=1, the Queue sends a message to each consumer at a time; after the consumer processes the message, the Queue will Send another message to the consumer.
Exchange
In the previous section we saw that the producer posts messages to the Queue, which in fact never happens in RabbitMQ. The actual situation is that the producer sends the message to the Exchange (exchange, X in the figure below), and the Exchange routes the message to one or more Queues (or discards). According to what logic does Exchange route messages to Queue? This will be introduced in the Binding section. There are four types of Exchange in RabbitMQ, and different types have different routing strategies, which will be introduced in the Exchange Types section.
routing key
When a producer sends a message to Exchange, it usually specifies a routing key to specify the routing rules for the message, and this routing key needs to be used in conjunction with the Exchange Type and the binding key to take effect.
In the case where the Exchange Type and the binding key are fixed (in normal use, these contents are generally fixedly configured), our producer can determine where the message flows by specifying the routing key when sending a message to the Exchange.
The length limit set by RabbitMQ for the routing key is 255 bytes.
Binding
In RabbitMQ, Exchange is associated with Queue through Binding, so that RabbitMQ knows how to correctly route messages to the specified Queue.
Binding key
When binding Exchange and Queue, a binding key is generally specified; when a producer sends a message to Exchange, a routing key is generally specified; when the binding key matches the routing key, the message will be routed to the corresponding Queue. This will be illustrated with practical examples in the Exchange Types chapter.
When binding multiple Queues to the same Exchange, these Bindings allow the same binding key to be used.
The binding key does not take effect in all cases. It depends on the Exchange Type. For example, a fanout type of Exchange will ignore the binding key and route messages to all Queues bound to the Exchange.
Exchange Types
The commonly used Exchange Types in RabbitMQ are fanout, direct, topic, and headers (there are also two Exchange Types mentioned in the AMQP specification, which are system and custom, which will not be described here).
fanout
The exchange routing rule of the fanout type is very simple, it will route all messages sent to the exchange to all the Queues bound to it. In the above figure, all messages sent by the producer (P) to the Exchange (X) will be routed to the two Queues in the figure, and eventually consumed by the two consumers (C1 and C2).
direct
The direct type Exchange routing rule is also very simple, it will route the message to those Queues whose binding key exactly matches the routing key. As an example of the configuration in the above figure, we send a message to Exchange with routingKey=”error”, and the message will be routed to Queue1 (amqp.gen-S9b…, which is the Queue name automatically generated by RabbitMQ) and Queue2 (amqp.gen- Agl...); if we send a message with routingKey=”info” or routingKey=”warning”, the message will only be routed to Queue2. If we send the message with other routingKey, the message will not be routed into these two Queues.
topic
As mentioned earlier, the direct-type Exchange routing rule completely matches the binding key and routing key, but this strict matching method cannot meet actual business needs in many cases. The topic-type Exchange has been extended on the matching rules. It is similar to the direct-type Exchage. It also routes messages to the Queue whose binding key matches the routing key. However, the matching rules here are somewhat different. It agrees:
- The routing key is a string separated by a period ". " (we call each independent string separated by a period ". " a word), such as "stock.usd.nyse", "nyse. vmw", "quick.orange.rabbit"
- The binding key, like the routing key, is also a string separated by a period "."
- There can be two special characters "*" and "#" in the binding key, which are used for fuzzy matching, where "*" is used to match one word, and "#" is used to match multiple words (can be zero)
Take the configuration in the above figure as an example, the message with routingKey=”quick.orange.rabbit” will be routed to Q1 and Q2 at the same time, and the message with routingKey=”lazy.orange.fox” will be routed to Q1 and Q2, routingKey=”lazy. The message of brown.fox" will be routed to Q2, and the message of routingKey=”lazy.pink.rabbit” will be routed to Q2 (it will only be delivered to Q2 once, although this routingKey matches both bindingKeys of Q2); routingKey=” quick.brown.fox”, routingKey=”orange”, routingKey=”quick.orange.male.rabbit” messages will be discarded because they do not match any bindingKey.
headers
The headers type of Exchange does not rely on the matching rule between the routing key and the binding key to route messages, but matches according to the headers attribute in the content of the sent message.
Specify a set of key-value pairs when binding Queue and Exchange; when a message is sent to Exchange, RabbitMQ will get the headers of the message (also in the form of a key-value pair), and compare whether the key-value pair exactly matches Queue and The key-value pair specified when the Exchange is bound; if it matches exactly, the message will be routed to the Queue, otherwise it will not be routed to the Queue.
This type of Exchange has not been used (but it should be very useful), so I will not introduce it.
RPC
MQ itself is based on asynchronous message processing. In the previous example, all producers (P) will not know the success or failure of the consumer (C) processing after sending the message to RabbitMQ (even whether there is a consumer to process the message). Do not know at all).
But in the actual application scenario, we may need some synchronous processing, and we need to wait for the server to finish processing my message before proceeding to the next step. This is equivalent to RPC (Remote Procedure Call). RPC is also supported in RabbitMQ. The mechanism for implementing RPC in RabbitMQ is:
- When the client sends a request (message), two values are set in the properties of the message (MessageProperties, 14 properties are defined in the AMQP protocol, these properties will be sent with the message) replyTo (a Queue name, used to tell the server After the processing is completed, the message that will notify me will be sent to this Queue) and correlationId (the identification number of this request. The server needs to return this attribute after processing is completed. The client will know which request was successfully executed or executed based on this id. fail)
- The server receives the message and processes it
- After the server finishes processing the message, it will generate a reply message to the Queue specified by replyTo, with the correlationId attribute at the same time.
- The client has previously subscribed to the Queue specified by replyTo. After receiving the reply message from the server, it analyzes which request has been executed according to the correlationId attribute, and performs subsequent business processing according to the execution result.
About transaction, confirm, ack
In the process of using RabbitMQ, you will definitely encounter such concepts: transaction, confirm, ack. This article introduces these concepts and the relationship between them.
RabbitMQ is the AMQP protocol used. The AMQP protocol defines "acknowledgement", which is the confirmation from the consumer to RabbitMQ, indicating that a message has been correctly processed by the client. RabbitMQ extends the AMQP protocol and defines a "confirm" from the broker to the publisher, but calls it confirm. So RabbitMQ has two kinds of confirmations, called different names, one is consumer acknowledgement, and the other is publisher confirm.
According to the AMQP protocol, the consumer acknowledgemenet is implemented by the basic.ack method. After receiving a message, the consumer can send the basic.ack method to the broker to confirm that a message has been received. By default, the consumer acknowledgement mode is enabled. If you do not want to send basic.ack, you can specify the no-ack parameter when sending the basic.consume method to disable the consumer acknowledgement mode.
Publisher confirm does not add a new confirmation method based on the AMQ protocol, but reuses the basic.ack method. But the publisher confirm mode is not turned on by default, you need to call the confirm.select method to set the channel to confirm mode. When the confirm mode is turned on, only when a message is accepted by all mirrors, the publisher will receive the confirmation of the message, which is a basic.ack method.
RabbitMQ supports transactions. Transaction mode is not enabled by default. You need to call the tx.select method to enable transaction mode. When transaction mode is enabled, tx.commit-ok will be returned to the client only after a transaction has been accepted by all mirrors. Both confirm mode and open transaction mode can be guaranteed to be "accepted by all mirrors", so what is the difference between open confirm mode and open transaction mode? The difference is that confirm is for one message, while transactions can be for multiple messages (of course, multiple messages for the same queue). In addition, the confirm mode is only for the settings of the publisher, and the transaction mode can be for the publisher or the consumer. If the transaction mode is set for the publisher, we can put multiple basic.publish methods in a transaction. When all the publish messages are accepted by all mirrors, the publisher client will receive the tx.commit-ok method. If the transaction mode is set for the consumer, we can put multiple basic.ack methods in a transaction, and when tx.commit-ok is received, it means that these messages have been confirmed.