In-depth understanding of RabbitMQ and AMQP protocols

Introduction

This article introduces RabbitMQ from the concepts of AMQP protocol (Advanced Message Queuing Protocol), message function, consumption model, financial-level usage and comparison of other function points, and hopes to help you understand RabbitMQ in depth.

AMQP protocol concept

The AMQP protocol itself defines many concepts. The following will analyze these concepts first, and will focus more on the scope, responsibilities, affiliation and other dimensions of each conceptual entity.

Connection

Corresponds to a TCP connection from an underlying AMQP-Client to RabbitMQ-Broker.
There are two endpoints to consider here. After the TCP connection is established, as shown in the figure below, the target Broker of the connection has been determined to be one in the cluster. Since it is a long connection, unless the connection is disconnected and rebuilt, the peer node will Immutable.
So it can be seen from here that RabbitMQ is different from Pulsar and RocketMQ in that it is a server-side addressing model. From the perspective of the Client, if you want to connect to any Exchange or Queue, you only need to connect to any Broker. Just do it.

Channel

Channels can be understood as a logical connection, reflecting the design idea of multiplexing.
Support serial execution, including receiving and sending instructions, can be understood as a "virtual network channel" in half-duplex mode.
All Exchange, Queue, and Binding operations are performed on Channel.

Vhost

It is equivalent to a concept of tenant isolation. Exchanges and Queues with the same name can be created under different Vhosts, so that business isolation can be performed.
The mechanism of permission isolation and permission dimension control of RabbitMQ is at the Vhost level.
Rabbit's official native global policy is controlled at the Vhost level.

Exchange

A virtual entity that declares routing policies for different messages and does not store messages itself.
A router that routes messages to specific Queues that meet the conditions based on the RoutingKey and Header in the message header.
Unicast and broadcast are supported.

Queue

The message storage entity is a container for the underlying storage of messages, similar to Pulsar's Topic.
In the single subscription mode, the consumers under it consume a part of the messages respectively.
It is associated with storage, so it has the characteristics of storage layer such as capacity upper limit and TTL.
Supports multiple consumption and exclusive consumption, depending on the parameters you set when you subscribe. Since it stores the message of the message system, it internally controls the progress of persistent consumption based on a message site, and records the position of the last consumed and Ack.
Consumer-oriented.

Binding

The bridge between Exchange and Queue is essentially a statement of rules.
An Exchange can have multiple Bindings.
A Queue can also be associated with multiple Bindings.
An Exchange to a Queue can also declare multiple Bindings.

message function

The following introduces the open source native functions officially provided by RabbitMQ. We know that the AMQP protocol can be regarded as a programmable message queue protocol. Based on the basic model provided by it, it can construct a variety of Diverse business models.

message structure

# publishInfo
exchange: amq.direct
immediate: false
mandatory: false
routingKey: test

# headerBody
bodySize: 1024
properties: 
    - contentType: 
    - encoding: 
    - deliveryMode: 
    - priority: 
    - correlationId: 
    - replyTo: 
    - expiration: 
    - messageId: 
    - timestamp: 
    - type: 
    - userId: 
    - appId: 
    - clusterId: 
    - headers: {}

# contentBody
二进制消息体bytes

Each message is divided into three parts, which are three independent data frames at the network level:

PublishInfo : Message routing declaration information, which can be associated with pre-declarations such as the target mailbox filled in when writing an email, whether to receive a receipt or not.
HeaderBody : The message header, which is used to store the pre-defined declaration of RabbitMQ itself. It can be the same as the Header of the HTTP protocol of the association layer. Some context information that is transparent to the business can be placed here to provide certain functions, such as distributed link tracking. TraceId.
ContentBody : message body, indistinguishable binary data block, the server does not know whether it is compressed, encrypted, etc., and only transparently stores and reads and delivers.

work queue

It is a simplification of the model. When sending a message, specify that Exchange is empty, and RoutingKey is QueueName. Broker will directly send the message to the target Queue in the future, which is equivalent to no Exchange for the user. He thinks that it is directly consumed by Queue. It's simpler.
Work queues are only suitable for single subscription scenarios, because Queue is only suitable for single subscriptions.

Official explanation

Publish/Subscribe publish-subscribe model

Queue does not support multiple subscriptions, which is achieved by converting ideas:

A Fanout-type Exchange: Equivalent to a Topic in a multi-subscription scenario.
Multiple different Queues: bound to the Exchange, equivalent to Subscription in a multi-subscription scenario.
Multiple Consumers consume the same Queue: multiple subscriptions in normal scenarios.
Each Consumer consumes a Queue: instance-level broadcast.

Official explanation

Align RocketMQ and Pulsar's multi-subscription consumption and broadcast consumption

Routing routing mode

Routing mode is the most commonly used mode in Rabbit.
Producer publishes an Exchange, the type of Exchange is Direct, specify RoutingKey in Message, and set a non-null value, and then declare some Queues. When declaring and binding Exchange, these Queues need to specify Binding, and the message is in When routing, determine whether the RoutingKey and BindingKey in the message are equal. If they are equal, they can be routed.
A message distribution scenario similar to tag filtering.

Official explanation

Align the tags of rocketmq and pulsar to filter consumption

Topic wildcard pattern

An upgraded version of routing mode, which supports wildcard matching.
Exchange type is Topic.
The matching rules are not regular expressions, but AMQP's own syntax.

Official explanation

Header mode

Not commonly used, matching rules are not based on RoutingKey, but on key-value pairs in HeaderBody.Properties.Headers.
Exact matching of all key-value pairs is supported.
Supports matching only one key-value pair.

RPC mode

RPC mode is not commonly used and is based on reply queues.
Producers and consumers use a question-and-answer model.
Equivalent to the request-response model of RPC.

Official explanation

consumption model

The consumption model is also a part of the use of a message system that requires special attention. In the process of business use, more attention will be paid to what a message has experienced in the entire process from production to delivery to consumers. The declaration cycle of the entire message is How to close the loop?

The following mainly analyzes the message life cycle of the RabbitMQ protocol from the implementation of the TDMQ RabbitMQ version.

View the consumption model from the life cycle of the message

Delivered but not Ack: Exclusively owned by the Consumer until the Consumer triggers an Ack or the Consumer disconnects.
Ack message: Marker consumed, site forward.
Nack message: The underlying operation is equivalent to Ack, and will be forwarded to the dead letter Exchange according to the configuration, otherwise discarded.
Requeue: The message is put back to the head of the queue for the next delivery.

Looking at the consumption model from the internal core components

Queue: responsible for storing the original message data, in order.
RedeliveryTracker: Responsible for recording the Requeue messages on the Consumer side, and triggering redelivery to identify the number of deliveries.
Dispatcher: Responsible for managing all Consumers connected to the Queue, responsible for load balancing, distribution, and progress management of messages.
Limiter: QoS current limiter, based on the number of Unacks instead of QPS, echoing the message life cycle above.
Unack Tracker: Tracks the messages that have been delivered but not acked in the current Channel.

What information can be obtained from this picture?

A Queue can be connected by different Connections and by different Channels of the same Connection.
Two Consumers can be started in a Channel to connect to the same Queue.
The QoS current limiting scope is Channel, that is, multiple Cconsumers created in a Channel enjoy the same quota.
If BasicQoS Global is set to true, the consumers in the same channel run out of quota, and all consumers under the channel are blocked and cannot receive new messages.
The Unack tracker is also the scope of the Channel, so a Channel is closed, and all the unacked messages that are exclusively owned by the Channel are recycled to the Queue-level tracker for global redelivery.

basic instruction set: https://www.rabbitmq.com/amqp-0-9-1-reference.html#basic.qos

Consumer Prefetch: https://www.rabbitmq.com/consumer-prefetch.html

Financial Grade Usage

Message Confirmation: Send feedback and give the Producer a confirmation that the message was sent successfully.
Alternative Exchange: The successfully sent message cannot match any Binding scenarios.
Message fallback: When the message fails to match any Binding, it falls back to the Producer.
Redelivery: In scenarios such as network errors, consumer downtime, and occasional business processing errors, retry consumption recovery.
Dead letter exchange: The business has been retried for many times and failed to succeed for a long time. It is put into a dead letter and waited for manual processing or the next automatic correction or alarm system.

Feature point comparison

After the above description, you should be able to use the function points of RabbitMQ to organize a relatively reasonable production and consumption topology based on your own business scenarios. In addition to the function points mentioned above, RabbitMQ itself also provides many other functions. The following mainly lists some comparisons for reference and reference:

channel class

function points	illustrate	TDMQ support
Authentication and Authorization	User/Password-based login authentication mechanism.	Integrate Pulsar's own JWT (Role+Token) mechanism for alignment
connection negotiation mechanism	The connection handshake negotiates connection communication parameters.	Fully aligned RabbitMQ native
Authentication and Authorization	The relationship between Vhost dimension configuration and User permissions.	AMQP SDK uses fully aligned layers
Current Limit Negotiation Mechanism (QoS)	Quota limits are based on the number of Unacks.	Fully aligned RabbitMQ native

Note: The implementation of the QoS mechanism RabbitMQ is different from the standard AMQP protocol. We choose to align RabbitMQ instead of the AMQP specification. We also believe that RabbitMQ's mode is more reasonable. See https://www.rabbitmq.com/consumer-prefetch for details. html

Exchange class

function points	illustrate	TDMQ support
ExchangeBindExchange	The extension of RabbitMQ on the AMQP protocol makes Exchange not limited to only binding to Queue, so that more complex topology logic can be constructed.	Not supported yet, scheduled
Dead Letter Exchange	The extended parameter of Queue, which is used to forward to the dead letter Exchange when the message is discarded in the Queue.	Fully aligned RabbitMQ native
Alternative Exchange	The extended parameter of Exchange is used for messages sent to Exchange that cannot match any routing rules to the downstream Queue and forwarded to the alternate Exchange.	Fully aligned RabbitMQ native

Queue class

function points	illustrate	TDMQ support
priority queue	Messages can be prioritized, and messages arriving at the same time can be delivered according to the priority, which is a function of locally destroying the first-in, first-out mechanism.	Not supported yet, scheduled
exclusive queue	The declared queue can only be connected by the declared Connection entity, usually used in conjunction with the temporary queue.	Not supported yet, scheduled
temporary queue	Randomly generates a temporary queue name, which can be used exclusively for the current process, usually used in conjunction with an exclusive queue and AutoDelete.	Not supported yet, scheduled
reply queue	It is used to declare the queue for returning packets to the Producer after the processing of the message Producer is completed, so as to realize the communication model of one question and one answer.	Not supported yet, scheduled
TTL	Set TTL (time to live) for messages, expired undelivered messages will be discarded or entered into dead letter.	Currently supports Vhost-level TTL mechanism
mirror queue	RabbitMQ was introduced to solve the problem of single-point storage, in order to realize multi-copy storage of queue messages.	TDMQ natural multi-copy distributed storage does not require this function

Transceiver Mechanism Class

function points	illustrate	TDMQ support
message confirmation	After the message is successfully stored in the Broker, it returns a packet to the Producer to confirm the successful sending.	Fully aligned RabbitMQ native
transaction message	The sending confirmation mechanism before the message confirmation function appears has poor performance and is not recommended.	Not yet supported, to be determined
delayed message	After the message is sent successfully, it is delayed for a certain period of time before delivery.	Fully aligned RabbitMQ native
RPC	Based on the one-question-answer model encapsulated by the reply queue, there are few usage scenarios, and it is recommended to use the mainstream RPC framework.	Not yet supported, to be determined

refer to

postscript

Through this article, I hope to have a certain degree of popular science about RabbitMQ, and from the perspective of designing a RabbitMQ Broker from 0 to 1, I briefly analyze some of the consumption model details of RabbitMQ, and add some information about this part on the current network. The lack of details may have some enlightening effects.

In the follow-up, we will focus on sharing how we built a set of high-performance, high-availability, cloud-native message queues that fully align with the RabbitMQ protocol in the apache pulsar ecosystem. Problems encountered, thinking generated.

Please look forward to it~