1. History
RabbitMQ is an open source implementation of AMQP (Advanced Message Queue) developed by erlang. The emergence of AMQP is actually in response to the needs of the general public. Although there are many open standards in the world of synchronous message communication (such as COBAR's IIOP, or SOAP, etc.), it is not the case in asynchronous message processing, only large enterprises There are some commercial implementations (such as Microsoft's MSMQ, IBM's Websphere MQ, etc.). Therefore, in June 2006, Cisco, Redhat, iMatix, etc. jointly formulated the open standard for AMQP.
RabbitMQ is developed by RabbitMQ Technologies Ltd and provides commercial support. The company was acquired by SpringSource (a division of VMWare) in April 2010. It was incorporated into Pivotal in May 2013. In fact, VMWare, Pivotal and EMC are essentially one company. The difference is that VMWare is an independent listed subsidiary, while Pivotal integrates certain EMC resources and is not currently listed.
RabbitMQ's official website is http://www.rabbitmq.com
2. Application scenarios
Closer to home. What problem does RabbitMQ, or AMQP solve, or what is its application scenario?
For a large software system, it will have many components or modules or subsystems or (subsystem or Component or submodule). So how do these modules communicate? This is very different from the traditional IPC. Many traditional IPCs are on a single system, and the modules are highly coupled and are not suitable for scalability. If sockets are used, different modules can indeed be deployed on different machines, but there are still many problems to be solved. such as:
1) How does the sender and receiver of the information maintain this connection? If one party's connection is interrupted, how will the data be lost during this period?
2) How to reduce the coupling between sender and receiver?
3) How to let the receiver with high Priority receive the data first?
4) How to achieve load balance? Effectively balance the receiver's load?
5) How to effectively send the data to the relevant recipients? In other words, subscribe the receiver to different data, how to do an effective filter.
6) How can it be extended, and even send this communication module to the cluster?
7) How to ensure that the receiver receives complete and correct data?
The AMDQ protocol solves the above problems, and RabbitMQ implements AMQP.
3. System architecture
image
RabbitMQ Server : Also called a broker server, it is not a truck that transports food, but a transmission service. The original word is RabbitMQisn't a food truck, it's a delivery service. His role is to maintain a route from Producer to Consumer to ensure that the data can be transmitted in the specified manner. But this guarantee is not a 100% guarantee, but it is enough for ordinary applications. Of course, for commercial systems, you can make another layer of data consistency guard, you can completely guarantee the consistency of the system.
Client P : Also called Producer, the sender of the data. createmessages and publish (send) them to a broker server (RabbitMQ). A Message has two parts: payload (payload) and label (label). As the name implies, the payload is the transmitted data. The label is the name of the exchange or a tag, which describes the payload, and RabbitMQ also uses this label to decide which Consumer to send this Message to. AMQP only describes the label, and RabbitMQ determines how to use the label.
Client C : Also called Consumer, the receiver of the data. Consumersattach to a broker server (RabbitMQ) and subscribe to a queue. Compare queue to a mailbox with a name. When a Message arrives at a mailbox, RabbitMQ sends it to one of its subscribers, the Consumer. Of course, the same Message may be sent to many Consumers. In this Message, only the payload and label have been deleted. For Consumer, it does not know who sent this information. It is that the protocol itself does not support it. But of course, it's another matter if the payload sent by the Producer contains the Producer's information.
For the correct transmission of a data from Producer to Consumer, there are three concepts that need to be clarified: exchanges, queues and bindings.
- Exchanges are where producers publish their messages.
- Queues are where the messages end up and are received by consumers
- Bindings are how the messages get routed from the exchange to particular queues.
There are several concepts that are not marked in the above figure, that is Connection (connection), Channel (channel, channel).
- Connection: It is a TCP connection. Both Producer and Consumer are connected to RabbitMQ Server via TCP. Later we can see that the beginning of the program is to establish this TCP connection.
- Channels: Virtual connection. It is established in the above-mentioned TCP connection. Data flow is carried out in Channel. In other words, the general situation is that the program starts to establish a TCP connection, and the second step is to establish this Channel.
So why use Channel instead of TCP connection directly?
For the OS, establishing and closing TCP connections comes at a price. Frequent establishment and closing of TCP connections has a great impact on the performance of the system, and the number of TCP connections is also limited, which also limits the system's ability to handle high concurrency. However, establishing a Channel in a TCP connection has no such cost. For Producer or Consumer, multiple channels can be used for Publish or Receive concurrently. Experiments have shown that 1s of data can publish 10K packets. Of course, for different hardware environments, this data must be different for different packet sizes, but I just want to explain that for ordinary Consumer or Producer, this is enough. If not enough, you should consider how to refine your design.
- Broker: Simply put, it is the message queue server entity.
- Exchange: Message exchange, which specifies the rules by which messages are routed to which queue.
- Queue: Message queue carrier, each message will be put into one or more queues.
- Binding: Binding, its role is to bind exchange and queue according to routing rules.
- Routing Key: routing key, exchange delivers messages based on this key.
- vhost: A virtual host. Multiple vhosts can be set up in a broker to separate the permissions of different users.
- producer: A message producer is a program that delivers messages.
- consumer: A message consumer is a program that receives messages.
- channel: Message channel. In each connection of the client, multiple channels can be established, and each channel represents a session task.
Exchange, Queue, and RoutingKey determine the only line from Exchange to Queue.
4. Basic concepts
ConnectionFactory、Connection、Channel
ConnectionFactory, Connection, and Channel are the most basic objects in the API provided by RabbitMQ. Connection is the socket link of RabbitMQ, which encapsulates the logic of the socket protocol. ConnectionFactory is the manufacturing factory for Connection.
Channel is the most important interface we deal with with RabbitMQ. Most of our business operations are completed in the interface of Channel, including defining Queue, defining Exchange, binding Queue and Exchange, and publishing messages.
Queue
Queue (Queue) is an internal object of RabbitMQ, used to store messages, represented by the following figure.
queue
image
The messages in RabbitMQ can only be stored in the Queue, the producer (P in the figure below) produces the message and finally delivers it to the Queue, and the consumer (C in the figure below) can get the message from the Queue and consume it.
image
Multiple consumers can subscribe to the same Queue. At this time, the messages in the Queue will be evenly distributed to multiple consumers for processing, rather than each consumer receiving all the messages and processing.
image
Message acknowledgment
In practical applications, it may happen that the consumer receives the message in the Queue, but it hangs down (or has other accidents) before the processing is completed. In this case, the message may be lost. In order to avoid this situation, we can ask consumers to send a receipt to RabbitMQ after consuming the message. RabbitMQ will remove the message from the Queue after receiving the message acknowledgment; Detecting that the consumer's RabbitMQ connection is disconnected, RabbitMQ will send the message to other consumers (if there are multiple consumers) for processing. There is no concept of timeout here. No matter how long a consumer processes a message, the message will not be sent to other consumers unless its RabbitMQ connection is disconnected.
There will be another problem here. If our developers forget to send the receipt to RabbitMQ after processing the business logic, this will lead to serious bugs-more and more messages will be accumulated in the Queue; consumers will restart Consume these messages repeatedly and execute the business logic repeatedly.
In addition, pub message has no ack.
Message durability
If we hope that even if the RabbitMQ service is restarted, the message will not be lost, we can set both Queue and Message to be durable (durable), which can ensure that in most cases our RabbitMQ messages will not be lost . But still can not solve the occurrence of a small probability loss event (for example, the RabbitMQ server has received the producer's message, but the RabbitMQ server will be powered off before it has time to persist the message), if we need to deal with this small probability event Management, then we have to use transactions. Since this is only a brief introduction to RabbitMQ, we will not explain RabbitMQ-related matters here.
Prefetch count
Earlier we said that if there are multiple consumers subscribing to the messages in the same Queue at the same time, the messages in the Queue will be shared among multiple consumers. At this time, if the processing time of each message is different, it may cause some consumers to be busy all the time, while others will quickly finish the work at hand and have been idle. We can set the prefetchCount to limit the number of messages that the Queue sends to each consumer each time. For example, if we set prefetchCount = 1, the Queue sends a message to each consumer each time; the Queue will Send another message to the consumer.
image
Exchange
In the previous section we saw that the producer posted the message to the Queue. In fact, this kind of thing never happens in RabbitMQ. The actual situation is that the producer sends the message to Exchange (Exchange, X in the figure below), and Exchange routes the message to one or more Queues (or discards).
image
What logic does Exchange use to route messages to Queue? This will be introduced in the Binding section.
There are four types of Exchange in RabbitMQ. Different types have different routing strategies, which will be introduced in the Exchange Types section.
routing key
When a producer sends a message to Exchange, it usually specifies a routing key to specify the routing rules of the message. This routing key needs to be used in conjunction with the Exchange Type and binding key to finally take effect.
In the case where the Exchange Type and binding key are fixed (generally these contents are fixed and configured in normal use), our producer can specify the routing key to determine where the message flows when sending a message to Exchange.
The length limit set by RabbitMQ for the routing key is 255 bytes.
Binding
In RabbitMQ, Exchange is associated with Queue through Binding, so RabbitMQ knows how to correctly route messages to the specified Queue.
image
Binding key
When binding (Binding) Exchange and Queue, a binding key is generally specified; when a consumer sends a message to Exchange, a routing key is generally specified; when the binding key matches the routing key, the message will be routed To the corresponding Queue. This will be illustrated with practical examples in the Exchange Types chapter.
When binding multiple Queues to the same Exchange, these Bindings allow the same binding key to be used.
The binding key does not take effect in all cases. It depends on the Exchange Type. For example, the Exchange of the fanout type will ignore the binding key, but will route the message to all Queues bound to the Exchange.
Exchange Types
There are four types of Exchange types commonly used in RabbitMQ: fanout, direct, topic, and headers (the AMQP specification also mentions two Exchange Types, which are system and custom, which are not described here), which are introduced below.
fanout
The Exchange routing rule of the fanout type is very simple. It will route all messages sent to the Exchange to all Queues bound to it.
image
In the figure above, all messages sent by the producer (P) to Exchange (X) will be routed to the two Queues in the figure, and will eventually be consumed by the two consumers (C1 and C2).
direct
The direct Exchange routing rule is also very simple, it will route the message to the Queue whose binding key exactly matches the routing key.
image
Taking the configuration in the above figure as an example, if we send a message to Exchange with routingKey = "error", the message will be routed to Queue1 (amqp.gen-S9b ..., which is the Queue name automatically generated by RabbitMQ) and Queue2 (amqp.gen- Agl ...); if we send the message with routingKey = "info" or routingKey = "warning", the message will only be routed to Queue2. If we send a message with another routingKey, the message will not be routed to these two Queues.
topic
As mentioned earlier, the Exchange routing rule of the direct type completely matches the binding key and the routing key, but this strict matching method cannot meet the actual business requirements in many cases. The topic type Exchange is extended in matching rules. It is similar to the direct type Exchage. It also routes messages to the Queue that matches the binding key and the routing key. However, the matching rules here are somewhat different. It agrees:
The routing key is a string separated by a period "." (we will call each independent string separated by a period "." a word), such as "stock.usd.nyse" and "nyse." "vmw", "quick.orange.rabbit" binding key and routing key are also character strings separated by period "."
There can be two special characters "" and "#" in the binding key, used for fuzzy matching, where "" is used to match one word, "#" is used to match multiple words (may be zero)
image
Taking the configuration in the above figure as an example, the message of routingKey = ”quick.orange.rabbit” will be routed to Q1 and Q2 at the same time, the message of routingKey = ”lazy.orange.fox” will be routed to Q1, routingKey = ”lazy.brown. "fox" message will be routed to Q2, routingKey = "lazy.pink.rabbit" message will be routed to Q2 (only delivered to Q2 once, although this routingKey matches both binding keys of Q2); routingKey = "quick. Brown.fox ", routingKey =" orange ", routingKey =" quick.orange.male.rabbit "messages will be discarded because they do not match any bindingKey.
headers
The Exchange of headers type does not depend on the matching rules of routing key and binding key to route the message, but matches according to the headers attribute in the content of the sent message.
Specify a set of key-value pairs when binding Queue and Exchange; when a message is sent to Exchange, RabbitMQ will get the headers of the message (also in the form of a key-value pair) and compare whether the key-value pairs in it match the Queue The key-value pairs specified during Exchange binding; if they match exactly, the message will be routed to the Queue, otherwise it will not be routed to the Queue.
This type of Exchange has not been used (but it should also be very useful), so I will not introduce it.
RPC
MQ itself is based on asynchronous message processing. In the previous example, all producers (P) send messages to RabbitMQ without knowing whether the consumer (C) processing succeeded or failed (even if there is a consumer to process this message) Do not know at all).
However, in the actual application scenario, we probably need some synchronization processing. We need to wait for the server to finish processing my message before proceeding to the next processing. This is equivalent to RPC (Remote Procedure Call, remote procedure call).
RPC is also supported in RabbitMQ.
image
The mechanism for implementing RPC in RabbitMQ is:
- When the client sends a request (message), set two values in the message properties (MessageProperties, defined in the AMQP protocol 14 properties, these properties will be sent along with the message) replyTo (a Queue name, used to tell the server After the processing is completed, the message that informs me will be sent to this Queue) and correlationId (the identification number of this request, the server needs to return this attribute after processing is completed, and the client will know which request was successfully executed or executed based on this id failure)
- The server receives the message and processes it
- After processing the message, the server will generate a response message to the Queue specified by replyTo, and bring the correlationId attribute
- The client has subscribed to the Queue specified by replyTo. After receiving the response message from the server, it analyzes which request was executed according to the correlationId attribute, and performs subsequent business processing based on the execution result.
5. Details
Use ack to confirm the correct delivery of Message
By default, if a Message has been correctly received by a Consumer, the Message will be removed from the queue. Of course, you can also send the same Message to many Consumers.
If a queue is not subscribed to by any consumer, then if data arrives for this queue, the data will be cached and not discarded. When there is a Consumer, this data will be sent to this Consumer immediately. When this data is correctly received by the Consumer, this data will be deleted from the queue.
So what is received correctly? Through ack. Each Message must be acknowledged (acknowledged, ack). We can display the ack in the program, or ack automatically. If there is data that has not been acked, then RabbitMQ Server will send this information to the next Consumer.
If the app has bugs and forgets the ack, then RabbitMQ Server will not send data to it again, because Server believes that this Consumer has limited processing power.
And the mechanism of ack can play the role of limiting current (Benefitto throttling): send ack after Consumer finishes processing the data, and even send ack after extra delay, it will effectively balance the load of Consumer.
Of course, for practical examples, for example, we may merge some data, such as the data in merge 4s, and then get the data after sleep 4s. Especially in monitoring the state of the system, we don't want all the state to be transferred in real time, but we want a certain delay. This can reduce some IO, and the end user will not feel it.
Reject a message
There are two ways, the first Reject allows RabbitMQ Server to send the message to the next Consumer. The second is to delete the Message immediately from the queue.
Creating a queue
Both Consumer and Procuder can create queues through queue.declare. For a Channel, Consumer cannot declare a queue, but subscribe to other queues. Of course, you can also create a private queue. Only the app itself can use this queue. The queue can also be deleted automatically. The queue marked as auto-delete will be deleted automatically after the last Consumer unsubscribe. So what if you are creating an existing queue? Then there will be no impact. It should be noted that there is no effect, that is to say, if the parameter is different from the first time in the second creation, although the operation is successful, the properties of the queue will not be modified.
So who should be responsible for creating this queue? Is it Consumer or Producer?
If the queue does not exist, of course, Consumer will not get any Message. But if the queue does not exist, the Producer Publish Message will be discarded. Therefore, in order not to lose data, both Consumer and Producer try to create the queue! Anyway, there will be no problems with this interface.
The queue's handling of load balance is perfect. For multiple consumers, RabbitMQ uses a round-robin method to send them to different consumers in a balanced manner.
Exchanges
As can be seen from the architecture diagram, Procuder Publish's Message entered Exchange. Then through "routing keys", RabbitMQ will find the queue in which this Message should be placed. The queue is also bound by the routing keys.
There are three types of Exchanges: direct, fanout, topic. Each implements a different routing algorithm.
- Direct exchange: If the routing key matches, then the Message will be delivered to the corresponding queue. In fact, when the queue is created, it will automatically use the queue name as the routing key to bind the exchange.
- Fanout exchange: Will broadcast to the response queue.
- Topic exchange: pattern matching on keys, such as ab can be passed to all ab queues.
Virtual hosts
Each virtual host is essentially a RabbitMQ Server, with its own queue, excagne, and bings rule, etc. This ensures that you can use RabbitMQ in multiple different applications.
The latest Java technology dry goods documentation: [Java core knowledge points finishing] covering 29 Java core technologies, JVM, Redis, Nginx, Spring Boot, Spring Cloud, Kafka, concurrent programming, Tomcat, MyBatis, BAT interview questions, Java Technical elaboration video, etc. Forward + follow, reply to "dry goods" by private message to get a free collection method.
Having mastered these knowledge points, you can get a lot of attention among candidates during the interview, critically hit 9999 points. Opportunities are reserved for those who are prepared. Only with sufficient preparation can one be able to stand out from the candidates.
