1 About message middleware
1.1 What is message middleware?
Message middleware refers to the basic software that completes the sending and receiving of messages in a distributed system.
Message middleware can also be called message queue (Message Queue / MQ), which uses an efficient and reliable message delivery mechanism for platform-independent data exchange, and integrates distributed systems based on data communication. Communication between processes can be scaled in a distributed environment by providing a message passing and message queue model.
In short, message middleware is often used in Internet scenarios to perform operations such as message routing, subscription publishing, and asynchronous processing to relieve system pressure.
1.2 What pain points does it solve for us?
1. Decoupling: For example, system A will hand over some things to system B, but A does not want to be directly related to B to avoid too strong coupling. By adding a message queue between A and B, A will be tasked Handed over to the message queue, B subscribes to the message queue to execute the task.
This scenario is very common. For example, A is an order system and B is an inventory system. The work of reducing the inventory can be handed over to the B system for processing through the message queue. If system A wants multiple systems B, C, D... to deal with the problem at the same time, this advantage is even more obvious.
2. Orderliness: First-in, first- out principle, first-come processing. For example, it takes a long time for a system to process something, but when processing this thing, other people also send requests, and the request can be placed in the message In the team, deal with it one by one.
For businesses that have strong requirements for the order and consistency of data, for example, the same bank card is used by multiple portals at the same time, it is necessary to ensure the order of incoming and outgoing accounts to avoid data inconsistency.
3. Message routing: send messages in the queue to different other queues according to different rules
Send different colored requests to different services for operation through message queues. This achieves the purpose of splitting traffic according to business.
4. Asynchronous processing: When processing a task, there are 3 steps A, B, and C. You need to complete the A operation first, and then do the B and C operations. The success of task execution depends strongly on the results of A, but not on the results of B and C.
If we use the serial execution method, the cycle of processing tasks will become longer, and the overall throughput of the system will also be reduced (doing asynchronous in the same system is actually a relatively large overhead), so it is better to use message queues way.
The login operation is a typical scenario: A: perform the login and get the result, B: record the login log, C: write the user information and Token into the cache. After executing A, you can jump from the login page to the home page. B and C allow the service to digest slowly without blocking the current operation.
5. Peak shaving: Reduce the operations during the peak period. For example, the entire operation process of classmate A consists of 12 steps. The subsequent 11 steps are data that do not need to focus on the results and can be placed in the message queue.
2 Execution principle of message middleware
2.1 Composition of message middleware
Broker: A message server that provides message core services as a Server.
Producer: message producer, the initiator of the business, responsible for producing messages and transmitting them to the broker,
Consumer: message consumer, the processor of the business, responsible for obtaining messages from the broker and processing business logic.
Topic: Topic, the unified collection of messages in the publish/subscribe mode, different producers send messages to the topic, and the MQ server distributes them to different subscribers to broadcast messages.
Queue: Queue, in PTP mode, a specific producer sends a message to a specific queue, and a consumer subscribes to a specific queue to receive the specified message.
Message: The message body is a data packet encoded according to a fixed format defined by different communication protocols to encapsulate business data and realize message transmission.
Take kafka as an example here. This is a typical cluster mode. Kafka manages the cluster configuration through Zookeeper, elects the leader, and performs rebalance when the Consumer Group changes. Producers use push mode to publish messages to brokers, and consumers use pull mode to subscribe and consume messages from brokers.
- The producer is responsible for producing messages
- consumer is responsible for consuming messages
- The broker message server, which provides the processing work of the message core
- zookeeper is used for producer and consumer registration and discovery
2.2 Pattern classification of message middleware
2.2.1 PTP point-to-point
Using the queue as the communication carrier, the message producer produces and sends the message to the queue, and then the message consumer takes the message from the queue and consumes the message.
After the message is consumed, it is no longer stored in the queue, so it is impossible for the message consumer to consume the message that has been consumed. Queue supports the existence of multiple consumers, but for a message, only one consumer can consume it.
Peer-to-peer mode consists of three roles:
- Message queue (Queue)
- Sender
- Receiver
Each message is sent to a specific queue, and the receiver gets the message from the queue. Queues hold messages, either in memory or persistent, until they are consumed or time out.
Features:
- There is only one consumer per message (that is, once consumed, the message is no longer in the message queue)
- There is no time dependency between sender and receiver
- The receiver needs to reply to the queue successfully after receiving the message successfully
- Using the first-in-first-out feature of FIFO, the order of messages can be guaranteed.
2.2.2 Pub/Sub publish and subscribe (broadcast)
Using a topic as a communication carrier, a message producer (publisher) publishes messages to a topic, and multiple message consumers (subscriptions) consume the message at the same time. Unlike the peer-to-peer approach, messages published to a topic are consumed by all subscribers.
The queue implements load balancing, and sends the messages produced by the producer to the message queue for consumption by multiple consumers. But a message can only be accepted by one consumer, when no consumer is available, the message will be held until there is an available consumer.
The publish-subscribe model consists of three roles:
- Topic
- Publisher
- Subscriber
Multiple publishers send messages to topics, and the system delivers these messages to multiple subscribers.
Features:
- Each message can have multiple consumers: unlike the peer-to-peer approach, published messages can be consumed by all subscribers
- There is a time dependency between publishers and subscribers.
- For a subscriber of a topic (Topic), it must create a subscriber before consuming messages from the publisher.
- In order to consume messages, subscribers must remain in a running state.
2.3 Advantages of message middleware
System decoupling: There is no direct calling relationship between interactive systems, but only through message transmission, so the system is not intrusive and has a low degree of coupling.
Improve system response time: For example, the original set of logic can put urgent and important (need to respond immediately) services to the calling method, and use message queues for low-response requirements, and put them in the MQ queue for consumers to process.
Provide services for big data processing architecture: through message as integration, in the context of big data, message queue is also integrated with real-time processing architecture to provide performance support for data processing.
2.4 Common Protocols for Message Middleware
Composition of AMQP protocol, MQTT protocol, STOMP protocol, XMPP protocol, and other TCP/IP-based custom protocol message middleware
2.4.1 AMQP
AMQP, the Advanced Message Queuing Protocol, is an open standard for application layer protocols, designed for message-oriented middleware. Message middleware is mainly used for decoupling between components, the sender of the message does not need to know the existence of the message consumer, and vice versa. The main features of AMQP are message-oriented, queuing, routing (including point-to-point and publish/subscribe), reliability, and security.
Advantages: Reliable and versatile.
2.4.2 MQTT
MQTT (Message Queuing Telemetry Transport, Message Queuing Telemetry Transport) is an instant messaging protocol developed by IBM, which may become an important part of the Internet of Things. The protocol supports all platforms and can connect almost all Internet-connected objects to the outside world, and is used as a communication protocol for sensors and actuators (such as connecting houses through Twitter).
Advantages: simple format, small bandwidth, mobile communication, PUSH, embedded system
2.4.3 STOMP
STOMP (Streaming Text Orientated Message Protocol) is a streaming text oriented message protocol, a simple text protocol designed for MOM (Message Oriented Middleware). STOMP provides an interoperable connection format that allows clients to interact with any STOMP message broker (Broker).
Advantages: command mode (non topic\queue mode)
2.4.4 XMPP
XMPP (Extensible Messaging and Presence Protocol) is a protocol based on Extensible Markup Language (XML) and is mostly used for instant messaging (IM) and online presence detection. For instant operations between servers. At its core, XML-based streaming is a protocol that may eventually allow Internet users to send instant messages to anyone else on the Internet,
even with different operating systems and browsers.
Advantages: universal and open, strong compatibility, extensibility, high security, but the XML encoding format occupies a large bandwidth
2.5 Mainstream message middleware
Common message middleware include: RabbitMQ, RocketMQ, kafka, etc. Later, we will have a special article to make a complete comparison of the capabilities and technical parameters of these middleware, and give technical selection suggestions in different business scenarios .
Original link:
https://www.cnblogs.com/wzh2010/p/15888498.htmlIf this article is helpful to you, please like and forward it