Article directory
Top Reasons to Design a Message Center
1. Realize system decoupling
Each subsystem or component communicates through the message center instead of calling directly, which can realize system decoupling and reduce the dependencies between components. This is conducive to the stability, scalability and maintainability of the system.
2. Asynchronous processing
The message center implements an asynchronous communication mechanism, which can help us realize asynchronous processing tasks or processes. After the producer delivers the message, it does not need to wait for the consumer's response, and continues to process other tasks. This helps improve processing efficiency.
3. Flow peak clipping
Message buffering can be realized by using the message queue, and messages are temporarily stored in the queue during the peak period, and the resources to be processed can be released before consumption. This helps to deal with high concurrency scenarios and prevent system crashes.
4. Automatic retry
When an exception occurs in consumption, the message will not be lost and will return to the queue. Consumers can re-consume periodically to implement an automatic retry mechanism. This helps improve system stability.
5. Sequence Guarantee
In some scenarios, the order of message consumption needs to be guaranteed. The message center uses the first-in-first-out feature of the queue to realize the orderly consumption of messages.
6. Message delivery guarantee
Using message queues, important messages in the production environment can be stored persistently. In this way, even if the consumer fails in the process of processing the message, the message will not be lost. After the consumer restarts, it can consume messages again, so as to ensure that the messages will not be missed.
7. Flow control
Through the message center, it is possible to control the rate of message production and consumption. When the downstream processing capacity cannot increase, rate limiting can be used to avoid message accumulation too fast; when the processing capacity is released, the rate can also be increased by accordion. This helps avoid system crashes.
In short, the main purpose of designing the message center is to achieve system decoupling, support asynchronous processing, peak shaving and current limit, automatic retry, sequence guarantee, reliable delivery of messages, and flow control. It is one of the core components to build a stable, reliable, flexible and scalable system architecture.
However, the introduction of the message center will also increase the complexity of the system, which requires strict management of the message transmission process to avoid message loss or repeated consumption. Therefore, whether a message center is needed or not needs to be weighed in combination with the specific system architecture and business scenarios.
Its specific typical design and implementation are as follows (here, taking RocketMQ as an example, a simple message center design and implementation is given):
1. Message format definition
First of all, it is necessary to define the general format and protocol of the message, which generally includes fields such as message ID, message type, message body, and timestamp. The format is usually JSON, XML or a custom binary format.
Define the message in JSON format:
{
"id": "msg_001",
"type": "ORDER_CREATE",
"content": {
"orderNo": "123456789",
"productCode": "P123456789"
},
"timestamp": 1577836800
}
2. Message sending
Provide APIs for message sending and receiving, which can be called by various business modules in the system. Send API generally needs to receive parameters such as message body, message type, receiver list, etc.
Use the sending API provided by RocketMQ:
Message msg = new Message("TOPIC1", "ORDER_CREATE", "{'orderNo':'12345'}".getBytes(RemotingHelper.DEFAULT_CHARSET));
SendResult sendResult = rocketMQProducer.send(msg);
3. Message receiving API
The receiving API generally returns a message queue or stream for consumption by business modules.
Use the receiving API provided by RocketMQ:
// 定义消费者
DefaultMQPushConsumer consumer = new DefaultMQPushConsumer("CID1");
// 设置消费者订阅的Topic和Tag
consumer.setNamesrvAddr("127.0.0.1:9876");
consumer.subscribe("TOPIC1", "*");
// 注册消费者监听并设置消费逻辑
consumer.registerMessageListener(new MessageListenerConcurrently() {
@Override
public ConsumeConcurrentlyStatus consumeMessage(List<MessageExt> msgs, ConsumeConcurrentlyContext context) {
for (MessageExt msg : msgs) {
// 消费逻辑
}
return ConsumeConcurrentlyStatus.CONSUME_SUCCESS;
}
});
// 启动消费者
consumer.start();
4. Message queue
To achieve asynchronous transmission and decoupling of messages, message queue middleware is generally used, such as Kafka, RabbitMQ, RocketMQ, etc. These message queues enable the pipelining and buffering of messages.
RocketMQ itself is a message queuing product, and its message queuing function will be used implicitly.
5. Message distribution
When the producer generates a message, it needs to implement a distribution mechanism to deliver the message to the designated receiver or receiving group. This needs to establish an association relationship between the message and the receiver, which is generally completed in the form of configuration.
Message distribution is realized through RocketMQ's Topic. The producer defines which Topic to send to, and the consumer subscribes to the Topic to obtain the message.
6. Message concurrency control
Considering that the generation speed of messages may far exceed the consumption speed in high-concurrency scenarios, a mechanism needs to be implemented to avoid system crashes caused by excessive accumulation of messages. A common way is to set the message queue size limit and rejection policy.
RocketMQ supports setting the number of queues and message size limits for each topic, which can be used to avoid excessive accumulation of messages.
7. Message reliability
In order to ensure that messages are not lost, the message center is usually required to implement mechanisms such as message persistence, message backtracking, and message reissue. Persistence is generally completed with a database; backtracking and reissue are implemented using the message history track in the message queue.
RocketMQ supports persistence and provides delay queue and dead letter queue functions to ensure that messages are not lost.
8. System monitoring
As a key basic component, the message center needs to provide a wealth of monitoring indicators and monitoring interfaces, including the monitoring of key indicators such as message generation rate, message accumulation, message consumption rate, number of connections, and delay.
RocketMQ provides a wealth of Dashboard and Metrics to monitor the rate, delay, accumulation and other indicators of message generation and consumption.
9. Rights Management
Access to the message center requires authority management, and different system modules and users should have different authority, for example, some modules can only send messages but not consume them. Rights management is generally accomplished through login authentication combined with configuration.
RocketMQ provides access control functions based on users, permissions, and roles, which can control users' read and write permissions on Topic.
The above is an example of a simple RocketMQ message center design and implementation. Using the complete services provided by RocketMQ, we can quickly build a message center solution to realize asynchronous decoupling and communication of various modules in the system. RocketMQ is just an option for many message queue products, and others such as Kafka and RabbitMQ also have all the functions to realize the message center. It can be selected and designed according to the business scenario.
write at the end
If you are interested in related articles, you can pay attention to the official account "Architecture Palace", and will continue to update AIGC, java basic interview questions, netty, spring boot, spring cloud and other series of articles, and a series of dry goods will be delivered at any time!