RabbitMQ: We usually talk about message queues, and we think of three of them: producers, consumers, and message queues. Producers send messages to message queues, and consumers get messages from message queues for processing. For RabbitMQ, it makes a layer of abstraction on this basis, and introduces the concept of exchange. The exchange is an intermediate bridge between producers and message queues. It plays a role in message routing, that is, It is said that the producer is not directly related to the message queue, but is sent to the exchange first, and then the exchange is routed to the corresponding queue. As for what rules it is routed to the message queue according to what we need to introduce below . The producer here does not directly send the message to the message queue, but sends the message to the Exchange by establishing a Channel (channel) with the Exchange (exchange), and the Exchange forwards the message to the specified message queue according to the routing rules. The message queue stores messages and waits for the consumer to retrieve the message. The consumer obtains the message from the message queue by establishing a Channel connected to the message queue.
AMQP: Advanced Message Queue, Advanced Message Queuing Protocol. It is an open standard of the application layer protocol, designed for message-oriented middleware. Clients and message middleware based on this protocol can transfer messages, and are not limited by products, development languages, and other conditions.
Implementation schematic:
Basic concepts:
1.Channel: An independent bidirectional data flow channel in a multiplexed connection. A channel is a virtual connection established within a real TCP connection, multiplexing the channel of the TCP connection.
2.Producer (message producer): A client application that publishes messages to a message queue.
3.Consumer (message consumer): The client application that gets the message from the message queue.
4.Message (message): A message consists of a message header and a message body. The message body is opaque, and the message header consists of a series of optional attributes, including routing-key (routing key), priority (message priority), delivery-mode (whether persistent storage), etc.
5. Routing Key: An attribute of the message header, which is used to mark the routing rules of the message and determines the forwarding path of the switch. The maximum length is 255 bytes.
6.Queue (message queue): A data structure that stores messages and is used to hold messages until they are sent to consumers. It is the container for the message and the destination for the message. A message can be put into one or more queues. The message has been in the queue, waiting for the consumer to connect to the queue to take the message. It should be noted that when multiple consumers subscribe to the same Queue, the messages in the Queue will be evenly distributed to multiple consumers for processing, instead of each consumer receiving all the messages and processing, each message only can be received by a subscriber.
7.Exchange (Exchange | Router): Provides matching between Producer and Queue, receives messages sent by producers and forwards these messages to message queues according to routing rules. The exchange is used to forward messages. It does not store messages. If there is no Queue bound to the Exchange, it will directly discard the messages sent by the Producer. The exchange has four message scheduling strategies (described below), namely fanout, direct, topic, and headers.
8.Binding (binding): used to establish the association between Exchange and Queue. A binding is a routing rule that connects Exchange and Queue based on Binding Key, so the exchange can be understood as a routing table composed of Bindings.
6.Binding Key: The binding relationship between Exchange and Queue, which is used to match the Routing Key. The maximum length is 255 bytes.
7.Broker: RabbitMQ Server, server entity.
The scheduling policy refers to the rules according to which Exchange forwards the message to one or more queues after receiving the message sent by the producer. The scheduling strategy is related to three factors: Exchange Type (the type of Exchange), Binding Key (the binding relationship between Exchange and Queue), and message marking information (Routing Key and headers). Exchange allocates messages according to the Routing Key of the message and the Binding Key of the Exchange-bound Queue. When a producer sends a message to Exchange, it generally specifies a Routing Key to specify the routing rules for the message, and this Routing Key needs to be used in conjunction with the Exchange Type and Binding Key to take effect. When the Exchange Type and Binding Key are fixed (generally these contents are fixedly configured), our producer can determine where the message flows by specifying the Routing Key when sending a message to the Exchange.
Exchange type introduction:
1, Fanout (subscription mode | broadcast mode)
Fanout exchange will route all messages sent to the exchange to all message queues bound to the exchange. The subscription mode has nothing to do with Binding Key and Routing Key. The exchange distributes the received message to all message queues with binding relationship (regardless of Binding Key and Routing Key). Similar to subnet broadcasts, each host in the subnet gets a copy of the message. Fanout switches are the fastest to forward messages.
2. Direct (routing mode)
The Direct switch needs the routing key of the message to exactly match the Binding Key between the Exchange and the Queue. If the match is successful, the message will be distributed to the Queue. Only when the Routing Key and the Binding Key match exactly, the message queue can get the message. Direct is the default mode for Exchange. RabbitMQ provides an Exchange by default, the name is an empty string, the type is Direct, and it is bound to all Queues (the Binding Key between each Queue and this unnamed Exchange is the name of the Queue). Therefore, sometimes we feel that we can send and receive messages without an exchange, but in fact we use the Exchange provided by RabbitMQ by default.
3. Topic (wildcard mode)
Topic exchange fuzzy matching according to regular expressions: use the message's Routing Key and the Binding Key between the Exchange and the Queue to perform fuzzy matching. If the match is successful, the message will be distributed to the Queue. The Routing Key is a string separated by a period ". " (we call each independent string separated by a period ". " a word). Binding Key, like Routing Key, is also a string separated by periods ". ". There can be two special characters "*" and "#" in the Binding Key, which are used for fuzzy matching, where "*" is used to match one word, and "#" is used to match multiple words (or zero or one). ).
For example, in the following message queue scenario, a topic type exchange is used:
RPC mechanism implementation:
MQ itself is based on asynchronous message processing. In the previous example, all producers (P) will not know the consumption after sending messages to RabbitMQ The user (C) succeeds or fails in processing, and does not even know if there are consumers to process the message. But in the actual application scenario, we may need some synchronous processing, and we need to wait for the server to finish processing my message before proceeding to the next step. This is equivalent to RPC (Remote Procedure Call).
The mechanism for implementing RPC in RabbitMQ is:
1. When the producer sends a request (message), two properties are set in the message properties (MessageProperties, 14 properties are defined in the AMQP protocol, and these properties will be sent with the message) The values replyTo (a Queue name, used to tell the consumer to send the notification message to this Queue after the processing is completed) and correlationId (the identification number of this request, this attribute needs to be returned after the consumer processing is completed, the producer will Know which request was executed successfully or failed according to this id).
2. The consumer receives the message and processes it.
3. After the consumer processes the message, it will generate a reply message to the Queue specified by replyTo, with the correlationId attribute.
4. The producer has previously subscribed to the Queue specified by replyTo. After receiving the reply message from the server, it analyzes which request has been executed according to the correlationId attribute, and performs subsequent business processing according to the execution result.
Message confirmation mechanism:
In practical applications, it may happen that the consumer receives the message in the Queue, but crashes (or other accidents) before the processing is completed. In this case, the message may be lost. In order to avoid this situation, we can ask the consumer to send a receipt to RabbitMQ after consuming the message, and RabbitMQ will remove the message from the Queue after receiving the message acknowledgment; if RabbitMQ does not receive the receipt and Detecting that the consumer's RabbitMQ connection is disconnected, RabbitMQ will send the message to other consumers (if there are multiple consumers) for processing. There is no concept of Timeout here, a consumer processing a message for a long time will not cause the message to be sent to other consumers unless its RabbitMQ connection is disconnected. Another problem will arise here. If our developers forget to send a receipt to RabbitMQ after processing the business logic, this will lead to serious problems. There will be more and more messages accumulated in the Queue, which will repeat after the consumer restarts. Consume these messages and execute business logic repeatedly. If we use the no-ack method for confirmation, that is, every time the Consumer receives the data, regardless of whether the processing is completed, RabbitMQ will immediately mark the Message as complete, and then delete it from the queue.
Message persistence mechanism:
If we hope that the message will not be lost even when the RabbitMQ service is restarted, we can set both Queue and Message to be durable, which can ensure that in most cases our RabbitMQ messages are not lost. But it still can't solve the occurrence of small probability loss events (for example, the RabbitMQ server has received the message from the producer, but the RabbitMQ server is powered off before it has time to persist the message), if we need to deal with such small probability events Managed, then we need to use affairs.
Transactions:
Support for transactions is an important feature of the AMQP protocol. Suppose that when the producer sends a persistent message to the server, because the consume command itself does not return any Response, even if the server crashes and the message is not persisted, the producer cannot know that the message has been lost. If you use a transaction at this time, that is, open a transaction through txSelect(), then send a message to the server, and then submit the transaction through txCommit(), you can guarantee that if txCommit() is committed, the message will be persistent, if If txCommit() has not been submitted, the server crashes, and the message will not be received by the server. Of course, Rabbit MQ also provides the txRollback() command to roll back a transaction.
Message distribution mechanism:
When we use the message system in the application, in general, the speed of the producer inserting data into the queue is relatively fast, but the consumer data consumption often involves some business logic processing, resulting in the speed not keeping up with the production of the producer. data. Therefore, if a producer corresponds to a consumer, it is easy to cause many messages to accumulate in the queue. At this point, you have to use the work queue. A queue has multiple consumers consuming data at the same time. Work queues have two ways to distribute data: round-robin and fair dispatch. Polling distribution: The queue sends the same amount of data to each consumer. Fair distribution: Consumers set to fetch a piece of data from the queue each time, and manually reply after consumption, and continue to fetch the next data from the queue.
1. Polling distribution
If there are two consumers in the work queue, and the amount of data obtained by the two consumers is the same, it will not cause the two consumers to obtain different amounts of data because the two consumers process data at different speeds. . However, there are some hidden dangers in this distribution method. Although the consumer gets the message, if the consumer fails to process the business logic successfully, the message does not exist in RabbitMQ. There will be cases where messages are lost and the business logic fails to process successfully.
2. Fair distribution
Consumers set to fetch a piece of data from the queue at a time, and turn off the automatic reply mechanism. After each piece of data is fetched, it will manually reply and continue to fetch the next piece of data. Different from polling distribution, when each consumer is set to only fetch one piece of data from the queue at a time, and turn off automatic response, manually send confirmation to the queue after each data is processed. In this way, the queue will send data to each message consumer fairly, consume one and then send the second, and you can see in the management interface that the data is reduced one by one as consumers consume it, not all at once. Done. With fair distribution, messages are not lost and business logic fails to process successfully.