Summary of key points of message queue

Summary of key points of message queue RabbitMQ

Summarize your views on message queues. Please correct me if I am wrong.

1.1. Three major characteristics of message queue

Decoupling : When an order calls inventory, it may start by specifying the number and type of parameters to be called. Later, when the inventory service interface is upgraded and maintained, the number or type of parameters to be called will change. In this way, the order service will also change accordingly, and the coincidence is too high. However, after the message queue is introduced, the order only needs to put the key information of the order into the message queue, and there is no need to worry too much about how to call the inventory service. The inventory can directly consume the message from the queue.

Asynchronous : place an order -> send a text message -> send an email. Generally, you place an order first and then send a text message or email. The execution time may include placing an order + entering the database + sending a text message + sending an email, etc. However, after the message queue is introduced, placing an order into the database and directly putting the order success information into the message queue takes only the time to enter the database and the queue, and there is no need to worry about subsequent operations. After receiving the messages from the queue, you can execute them separately.

Peak clipping : 1. After receiving the user's request, the server first writes it into the message queue. If the length of the added message queue exceeds the maximum value, it will directly abandon the user's request or jump to the error page. 2. The flash sale business performs subsequent processing based on the request information in the message queue.

1.2. Message queue asynchronous call

Both message queue and feign can make remote calls, but when feign makes multiple remote calls and performs CompletableFuture asynchronous orchestration, there will be problems with the cookie information in the request header not being shared and the context not being shared. For example, a remote call obtains the shopping cart service based on user information and generates a new thread. If it is serial, one thread will not lose the request header. If the new thread calls the request header and the request header is not shared, an error will be reported.

In message queues, messages are usually delivered in the form of message headers and message bodies . The message header contains some metadata information, such as the unique identifier of the message, the sending time of the message, etc. This metadata information will not be lost during the message delivery process, ensuring the integrity and traceability of the message.

1.3. Distributed transactions

When distributed transactions use seata, the TCmanage transaction coordinator will incur additional overhead in coordinating and isolating transactions. Compared with the reliability of message queues, message delivery will be slower. Message queues rely on asynchronous performance to improve performance. When stronger transaction management capabilities and consistency guarantees are required, using Seata may be a more suitable choice, although it may be slightly slower. In scenarios with high performance requirements and low transaction management requirements, it may be more suitable to directly use the message queue interface.

1.3.1. Message queue reliability message delivery

ConfirmCallbackThe message queue uses an eventual consistency solution. For reliability, two callback functions can be set when the message is pushed to the switch and the switch pushes it to the queue. When the RetrunCallbackmessage is accepted, ack and nack can be set to be manually accepted or rejected.

Final consistency is based on the reliable delivery of messages. Due to delays or failures in messages due to network failures and other reasons, RabbitMQ can also ensure the final consistency of messages through mechanisms such as retry and persistence.

Message persistence: RabbitMQ allows messages to be persisted to disk so that messages can be recovered and re-delivered even after node failure or restart. This ensures that messages will not be lost due to node failure.

1.4. Elements of message queue

Messages are bound to the exchange through a routing key, and the exchange also has a binding key (routing key) bound to the queue. As for which queue the message will be sent to, it is matched by comparing the routing key of the message sent with the routing key of the queue bound to the exchange. Binding keys can use # to match multiple words, and * to match one word.

Switches are divided into one-to-one and one-to-many, one-to-one direct, one-to-many fanout sector switches, and topic switches. A sector switch will send messages to all queues bound to it (the routing key does not work here), and a topic switch will send messages to multiple queues with routing keys that pass specific binding rules. Generally, topics are used for one-to-many purposes.

Generally use @Configuration and @bean to annotate registered exchangers, queues, dead letter messages, etc.

1.5. Message loss, message idempotence, message sequential consumption, and message delayed consumption

• Message loss reference reliability message delivery
• Message idempotence Common methods to ensure idempotence in message queues include the following:

1. Unique identifier: Add a unique identifier (such as UUID) to the message. When processing the message, the consumer first checks whether the identifier has already processed the message. If it has been processed, it is ignored, otherwise the processing logic is executed.
2. Idempotence check: When processing a message, the consumer first queries the relevant data or status to determine whether the message has been processed. If it has been processed, success will be returned directly; if it has not been processed, the processing logic will be executed and the processing results will be recorded so that the next query can determine whether it has been processed.
3. Optimistic locking: When a consumer processes a message, optimistic locking is used to ensure that the same message is only processed once. The consumer acquires the lock before processing the message. If the acquisition is successful, the processing logic is executed and the lock is released after the processing is completed. If the acquisition fails, it means that the message has been processed by other consumers and the current consumer can ignore the message. .
4. Message deduplication: When processing a message, the consumer records the unique identifier of the message that has been processed. When receiving the same message next time, it will first determine whether it has been processed. If it has been processed, the message will be ignored directly to avoid Repeat process.

To sum up, ensuring the idempotence of the message queue needs to be processed on the consumer side by adding unique identifiers, idempotence checks, optimistic locking and message deduplication to ensure that the same message is only processed once.

• Message sequential consumption

1. Single consumer: Use a single consumer to consume messages, which ensures that messages are consumed in the order they are sent. This method is suitable for scenarios with high requirements on the order of message processing, but the disadvantage is that it cannot achieve parallel processing of messages.
2. Message partitioning: messages are partitioned according to certain rules, and each partition is consumed by a consumer. Through a reasonable partitioning strategy, it can be ensured that messages in the same partition are consumed in order, but the order of messages between different partitions cannot be guaranteed.
3. Message grouping: Group messages into groups, and each group is consumed by a consumer. Within the same group, messages can be consumed in order, but the order of messages between different groups cannot be guaranteed. Related messages can be placed in the same group according to business needs to achieve sequential consumption of messages.
4. Message reordering: Reorder messages on the consumer side. Consumers can cache received messages and then process them in a certain order. For example, you can use a priority queue or an ordered queue to achieve orderly consumption of messages.

Then proceed in a certain order. For example, you can use a priority queue or an ordered queue to achieve orderly consumption of messages.

• Delayed consumption: Delayed queues can be used, such as a switch and two queues. The switch is bound to the two queues and sets the expiration time of all messages in one of the queues . Once the message expires (dead letter), it is placed in Another queue, this time the consumer only needs to accept and consume another queue in real time.