Article Directory
1 Introduction
AMQP, or Advanced Message Queuing Protocol, is an open standard for application layer protocols designed for message-oriented middleware. Message middleware is mainly used for decoupling between components, and the sender of the message does not need to know the existence of the message user, and vice versa. The main features of AMQP are message-oriented, queue, routing (including point-to-point and publish/subscribe), reliability, and security.
The emergence of AMQP is actually in response to the needs of the general public. Although there are many open standards in the world of synchronous message communication (such as COBAR's IIOP, or SOAP, etc.), this is not the case in asynchronous message processing. Only large enterprises There are some commercial implementations (such as Microsoft's MSMQ, IBM's Websphere MQ, etc.), so in June 2006, Cisco, Redhat, iMatix, etc. jointly formulated the open standard of AMQP.
RabbitMQ is an open source AMQP implementation. The server side is written in Erlang language and supports multiple clients, such as: Python, Ruby, .NET, Java, JMS, C, PHP, ActionScript, XMPP, STOMP, etc., and supports AJAX. It is used to store and forward messages in a distributed system, and performs well in terms of ease of use, scalability, and high availability.
RabbitMQ is developed and commercially supported by RabbitMQ Technologies Ltd. The company was acquired by SpringSource (a division of VMWare) in April 2010. Merged into Pivotal in May 2013. In fact, VMWare, Pivotal and EMC are essentially one family. The difference is that VMWare is an independent listed subsidiary, while Pivotal integrates some resources of EMC and is not listed now.
RabbitMQ official website: http://www.rabbitmq.com
The following will focus on the basic concepts in RabbitMQ. Understanding these concepts is the prerequisite for using RabbitMQ well.
2. Basic objects
ConnectionFactory, Connection, and Channel are the most basic objects in the API provided by RabbitMQ.
ConnectionFactory is the manufacturing factory of Connection.
Connection is the socket link of RabbitMQ. It encapsulates the logic related to the socket protocol. It establishes a TCP connection, and both the producer and the consumer are connected to the RabbitMQ Server through TCP.
Channel is the most important interface for dealing with RabbitMQ. Most of the business operations are done in the Channel interface, including defining Queue, defining Exchange, binding Queue and Exchange, publishing messages, etc. Channel is a virtual connection, based on the above TCP connection, data flow is carried out through Channel. Why not directly establish data flow on the basis of TCP? If the data flow is established on the basis of TCP, there is a price for establishing and closing the TCP connection. Frequent establishment and closing of TCP connections has a great impact on the performance of the system, and the number of TCP connections is also limited, which affects the high concurrency capability of the system.
The following figure shows the architecture diagram of RabbitMQ:
3. Queue
Queue (queue) is an internal object of RabbitMQ for storing messages.
The messages in RabbitMQ can only be stored in the Queue, the producer produces the message and finally delivers it to the Queue, and the consumer can get the message from the Queue and consume it.
Multiple consumers can subscribe to the same Queue. At this time, the messages in the Queue will be evenly distributed to multiple consumers for processing, instead of each consumer receiving and processing all the messages. We will introduce ACK later .
In order to facilitate subsequent understanding, first introduce the following two concepts:
- Routing Key : Specifies who receives the current message.
- Binding Key : Specifies what RoutingKey will be assigned to the currently bound Queue under the current Exchange
3.1 Basic properties of Queue
- Type
The type of this Queue, the default is the Classic main queue, and it can also be set as a Quorum slave queue
- Name
Mandatory item, the name of the queue, the suggested format can be multiple fields, indicating the content stored in the queue, such as task.queue
- Durability
Whether persistence is required, there are two options Durable (persistent) and Transient (temporary)
- Auto delete
Whether to delete automatically, if you choose Yes, the message will be consumed by one of the consumers, then the queue will be automatically destroyed, and other consumers will also disconnect (the queue is gone, the connection must be broken).
- Arguments
RabbitMQ has many parameters that can be set, here is a brief introduction:
x-message-ttl: The value is a Number type, indicating the survival time of the message in the queue, in milliseconds
x-expires: The value is a Number type, indicating the time that the queue exists, in milliseconds
x-max-length: The value is Number type, the maximum number of messages that can be stored in a queue, and messages will be discarded from the head after the set number is exceeded. That is, the message that arrives first will be discarded first.
x-max-length-bytes: The value is String type, the maximum number of bytes for each message in the queue, and messages will be discarded from the head after the number exceeds the set number. That is, the message that arrives first will be discarded first.
x-overflow: The value is String type, which sets the behavior when the queue overflows, and the optional value is drop-head or reject-publish
x-dead-letter-exchange: The value is String type, if the message is rejected or expired, the name of the exchange where the message will be re-entered
x-dead-letter-routing-key: The value is String type. When the message is dead-lettered, the message will be routed according to the routing key. If not set, the original routing key of the message will be used
x-max-priority: The value is Number type, the maximum message priority, if not set, the message priority is not supported
x-queue-mode: The value is String type. If it is set to lazy, as many messages as possible will be saved to disk to reduce memory usage. If not set, all messages will be stored in memory to ensure the fastest distribution
x-queue-master-locator: The value is String type. When in the cluster, setting the queue to master location mode will determine the position of the queue master in the cluster
3.2 Message Acknowledgment Mechanism (ACK)
In practical applications, it may happen that the consumer receives the message in the Queue, but crashes (or has other accidents) before the processing is completed. In this case, the message may be lost. In order to avoid this from happening, we can require the consumer to send a receipt to RabbitMQ after consuming the message, and RabbitMQ will remove the message from the Queue only after receiving the message receipt.
If RabbitMQ does not receive a receipt and detects that the consumer's RabbitMQ connection is disconnected, RabbitMQ will send the message to other consumers (if there are multiple consumers) for processing. There is no concept of Timeout here. No matter how long a consumer processes a message, it will not cause the message to be sent to other consumers unless its RabbitMQ connection is disconnected.
There will be another problem here. If our developers forget to send a receipt to RabbitMQ after processing the business logic, this will cause serious bugs, and more and more messages will be accumulated in the Queue, and the consumer will repeat after restarting. These messages are consumed and the business logic is repeatedly executed.
3.3 Prefetch count
As mentioned above, if multiple consumers subscribe to the messages in the same Queue at the same time, the messages in the Queue will be shared among multiple consumers. At this time, if the processing time of each message is different, it may cause some consumers to be busy all the time, while other consumers will quickly finish the work at hand and remain idle. We can limit the number of messages Queue sends to each consumer by setting prefetchCount. For example, if we set prefetchCount=1, Queue will send a message to each consumer each time; after the consumer finishes processing this message, Queue will Send another message to the consumer.
4. Exchange
The message generated by the producer is not directly sent to the message queue Queue, but passes through Exchange, which then routes the message to one or more Queue, and also discards the message that does not meet the routing rules, which is different from the following The Exchange Type to be introduced is related.
How does Exchange accurately push messages to the corresponding Queue? RabbitMQ links Exchange and Queue together through Binding, so that Exchange knows how to accurately push messages to Queue. When binding (Binding) Exchange and Queue, a Binding Key is generally specified. When the producer sends a message to Exchange, a Routing Key is generally generated. When the Routing Key and Binding Key correspond, the message will be Send to the corresponding Queue.
4.1 Basic properties of Exchange
Since Durability and Auto delete have been introduced in the previous Queue, I will not repeat them here. The other modules are introduced below:
- exchange
The name of the switch.
- type
There are four common types of switches: fanout, direct, topic, and headers. Since the attribute Exchange Type has a lot of content, it will be introduced in detail below.
- Internal
Whether the Internal setting is used internally by RabbitMQ, the default is false. If it is set to true, it means that it is a built-in exchange, and the client program cannot directly send messages to this exchange, but can only be routed to the exchange through the exchange.
- Arguments
Extended parameters, used to extend the customized usage of AMQP protocol.
alternate-exchange: Set the "Alternate Exchange" parameter, if it cannot be routed to the exchange, send it to the specified alternate exchange.
4.2 Exchange Type
There are four types of Exchange, and different types have different strategies. Different types determine the bound Queue is different, the producer sends a message, the rule of Routing Key is A, then the producer will push the message with Routing Key=A to Exchange, at this time Exchange will have its own rules, The corresponding rules are used to filter the messages sent by the producers, and if the internal rules of Exchange can be met, the messages will be pushed to the corresponding Queue.
Comparison of 4 Exchange Types:
| type name | type description |
|---|---|
| fanout | Route all messages sent to the Exchange to all Queues bound to it |
| direct | Routing Key==Binding Key |
| topic | Fuzzy matching between Routing Key and Binding Key |
| headers | Messages are routed without relying on the matching rules between routing key and binding key, and are matched according to the headers attribute in the content of the sent message. |
The types of Exchange are explained in detail below.
- fanout
Broadcast, fanout will route all messages sent to the Exchange to all Queues bound to it.
Applicable scenarios :
-
Large players can use it to broadcast big news while playing online games.
-
Sports news is updated to the mobile client in real time.
-
Group chat function, broadcast message to everyone in the current group chat.
-
direct
Orientation, it will route the message to the Queue whose binding key exactly matches the routing key.
Applicable scenarios :
-
This type of Exchange usually distributes the same message to different Queues in a circular manner, that is, different consumers. Using this method, it is worth noting that the message is balanced among consumers. , instead of saying that the message is balanced between Queues.
-
topic
As mentioned above, the exchange routing rules of the direct type completely match the binding key and the routing key, but this strict matching method cannot meet actual business needs in many cases.
The topic-type Exchange has been extended on the matching rules. It is similar to the direct-type Exchange. It also routes messages to the Queue whose binding key matches the routing key.
Wildcard matching rules are as follows:
- Routing Key is a period- separated string (we call each independent string separated by a
.period a word ), such as "stock.usd.nyse", "nyse.vmw", "quick..orange.rabbit” .Binding Key is also a period -separated string like routing key*There can be two special characters and in the Binding Key#, used for fuzzy matching, which*is used to match one word and#used to match multiple words (can be zero)
Topic Exchange sends messages to one or more Queues according to the Routing Key and Exchange type. We often use it to implement various publish/subscribe, that is, publish and subscribe, which is also the ExchangeType we often use.
Applicable scenarios :
-
Category Updates for News
-
Agree that tasks are coordinated by multiple workers
-
The same problem needs specific people to know
-
headers
The difference between header mode and routing is that the header mode cancels the Routing Key and uses the k/v key-value pair in the header to match the queue.
Specify a set of key-value pairs when binding Queue and Exchange. When a message is sent to Exchange, RabbitMQ will get the headers of the message (also in the form of a key-value pair), and compare whether the key-value pair exactly matches Queue and Exchange. The key-value pair specified when Exchange is bound; if there is an exact match, the message will be routed to the Queue, otherwise it will not be routed to the Queue.
This type of Exchange has never been used, so it will not be introduced for the time being. If it is used in depth later, it will be introduced in detail.
5. Supplements to other related concepts
- Broker : message queue service process, this process includes two parts: Exchange and Queue
- Producer : The message producer, that is, the producer client, the producer client sends the message
- Consumer : The message consumer, that is, the consumer client, receives the message forwarded by MQ.
Producer sends message process:
- The producer establishes a TCP connection with the Broker.
- Producers and Brokers establish channels.
- The producer sends the message to the Broker through the channel, and the Exchange forwards the message.
- Exchange forwards the message to the specified Queue (queue)
Consumer receiving message process:
- The consumer establishes a TCP connection with the Broker
- Consumers and Brokers establish channels
- Consumers listen to the specified Queue (queue)
- When a message arrives on the Queue, the Broker pushes the message to the consumer by default.
- The consumer receives the message.
- ack reply
reference article
- https://blog.csdn.net/kavito/article/details/91403659
- https://blog.csdn.net/dandanzmc/article/details/52262850
- https://www.cnblogs.com/williamjie/p/9481774.html
- https://www.cnblogs.com/panchanggui/p/10330064.html
- https://blog.csdn.net/ad132126/article/details/83539213
- https://www.cnblogs.com/yaochc/p/9273321.html
- https://blog.csdn.net/a972669015/article/details/117629155