1. Knowledge of message middleware
1 Overview
Personal understanding is a tool for message transmission. It is generally used in systems with a relatively large amount of data to reduce the degree of coupling between systems and focus on the sending and receiving of data.
2. Composition of message middleware
2.1 Broker
Message server, as a server to provide core message services
2.2 Producer
The message producer, the initiator of the business, is responsible for producing the message and transmitting it to the broker,
2.3 Consumer
The message consumer, the business processor, is responsible for obtaining messages from the broker and processing business logic
2.4 Topic
Topic, the unified collection of messages in the publish-subscribe mode, different producers send messages to the topic, which are distributed by the MQ server to different subscribers to realize the broadcast of the message
2.5 Queue
Queue, in PTP mode, a specific producer sends a message to a specific queue, and the consumer subscribes to a specific queue to complete the reception of the specified message
2.6 Message
The message body is a data packet encoded according to a fixed format defined by different communication protocols to encapsulate the business data and realize the transmission of the message
3. Pattern classification of message middleware
3.1点对点模式
That is, each message can only be consumed once, a message can have multiple consumers, but a message can only be consumed by one consumer.
Implementation principle:
using queue (Queue) implementation, the message producer produces the message and sends it to the queue, and then the message consumer takes it out of the queue and consumes the message.
3.2发布/订阅模式
That is, each message can be consumed by multiple consumers, using topic as a communication carrier.
Implementation principle:
Different producers send messages to the topic, which are distributed by the MQ server to different subscribers, to realize the broadcast of the message, so that each consumer can get a copy of the message.
4. Advantages of message middleware
4.1系统解耦。
Reduce the degree of coupling between systems, facilitating system expansion and maintenance
4.2 提高系统的相应时间
Separate the less demanding ones and use the queue for processing, so that the system can focus on urgent business.
4.3数据持久化
Using message middleware can effectively prevent data loss.
5. Application scenarios of message middleware
5.1异步通讯
Some businesses do not want or need to process messages immediately. The message queue provides an asynchronous processing mechanism that allows users to put a message into the queue, but does not process it immediately. Put as many messages as you want in the queue, and then process them when needed.
5.2系统解耦
Reduce the degree of strong dependence between projects and adapt to heterogeneous systems. At the beginning of the project, it is extremely difficult to predict what needs the project will encounter in the future. An implicit data-based interface layer is inserted into the processing process through the message system. The processing on both sides must implement this interface. When the application changes, the processing on both sides can be independently extended or modified, as long as you ensure They follow the same interface constraints.
5.3数据持久化（冗余数据）
In some cases, the process of processing data will fail. Unless the data is persisted, it will be lost. The message queue persists the data until they have been completely processed, avoiding the risk of data loss in this way. In the "insert-get-delete" paradigm adopted by many message queues, before deleting a message from the queue, your processing system needs to clearly indicate that the message has been processed to ensure that your data is stored safely Until you finish using it.
5.4 扩展性
Because the message queue decouples your processing process, it is easy to increase the frequency of message enqueuing and processing, as long as you increase the processing process. No need to change the code, no need to adjust the parameters. Facilitate distributed expansion.
5.5过载保护
In the case of a surge in traffic, applications still need to continue to function, but such sudden traffic cannot be predicted; it is undoubtedly a huge waste to invest resources on standby at any time in order to be able to handle such instant peak visits as the standard. The use of message queues enables key components to withstand sudden access pressure, and will not completely collapse due to sudden overloaded requests.
5.6 可恢复性
When some components of the system fail, the entire system will not be affected. The message queue reduces the coupling between processes, so even if a message processing process hangs up, the messages added to the queue can still be processed after the system is restored.
5.7顺序保证
In most usage scenarios, the order of data processing is important. Most message queues are sorted by nature, and it is guaranteed that the data will be processed in a specific order.
5.8缓冲
In any important system, there will be elements that require different processing time. The message queue helps the most efficient execution of tasks through a buffer layer, which helps to control and optimize the speed of data flow through the system. To adjust the system response time.
5.9 数据流处理
Mass data streams generated by distributed systems, such as business logs, monitoring data, user behaviors, etc., are collected and summarized in real time or in batches for these data streams, and then big data analysis is a must-have technology for the current Internet. This is done through message queues. Class data collection is the best choice.
6. Commonly used protocols for message middleware
6.1 AMQP协议
AMQP is Advanced Message Queuing Protocol, an application layer standard advanced message queuing protocol that provides unified messaging services. It is an open standard for application layer protocols and is designed for message-oriented middleware. Clients and message middleware based on this protocol can transmit messages, and are not restricted by different client/middleware products, different development languages and other conditions.
Advantages: reliable and universal
6.2 MQTT协议
MQTT (Message Queuing Telemetry Transport) is an instant messaging protocol developed by IBM and may become an important part of the Internet of Things. This protocol supports all platforms and can connect almost all networked objects to the outside world, and is used as a communication protocol for sensors and actuators (such as connecting houses via Twitter).
Advantages: simple format, small occupied bandwidth, mobile communication, PUSH, embedded system
6.3 STOMP协议
STOMP (Streaming Text Orientated Message Protocol) is a simple text protocol designed for MOM (Message Oriented Middleware). STOMP provides an interoperable connection format that allows clients to interact with any STOMP message broker (Broker).
Advantages: command mode (non-topic\queue mode)
6.4 XMPP协议
XMPP (Extensible Messaging and Presence Protocol) is a protocol based on Extensible Markup Language (XML), which is mostly used for instant messaging (IM) and online field detection. It is suitable for quasi-real-time operation between servers. The core is based on XML streaming. This protocol may eventually allow Internet users to send instant messages to anyone else on the Internet, even if their operating systems and browsers are different.
Advantages: universal openness, strong compatibility, scalability, and high security, but the XML encoding format takes up a lot of bandwidth
6.5 其他基于TCP/IP自定义的协议
Some special frameworks (such as redis, kafka, zeroMq, etc.) do not strictly follow the MQ specification according to their own needs. Instead, they encapsulate a set of protocols based on TCP\IP and transmit through the network socket interface to realize the function of MQ.
7. Introduction to common message middleware
7.1RocketMQ
A distributed, queue model message middleware under the Alibaba Department, formerly known as Metaq, and the name of the 3.0 version was changed to RocketMQ. It is a set of mq implemented by Alibaba based on the design philosophy of Kafka using java.
7.2RabbitMQ
An open source message queue written in Erlang supports many protocols: AMQP, XMPP, SMTP, STOMP. This is also true, making it a very heavyweight and more suitable for enterprise-level development. At the same time, the Broker architecture is implemented. The core idea is that the producer will not send the message directly to the queue, and the message will be queued in the central queue when it is sent to the client. It has good support for routing, load balance, and data persistence. It is mostly used for enterprise-level ESB integration.
7.3ActiveMQ
A sub-project under Apache. Using Java to fully support JMS 1.1 and J2EE 1.4 standard implementation of JMS Provider, a small amount of code can efficiently implement advanced application scenarios.
7.4Redis
A Key-Value NoSQL database developed in C language is very active in development and maintenance. Although it is a Key-Value database storage system, it supports the MQ function itself, so it can be used as a lightweight queue service. The enqueue and dequeue operations of RabbitMQ and Redis are executed 1 million times each, and the execution time is recorded every 100,000 times. The test data is divided into four different sizes of 128Bytes, 512Bytes, 1K and 10K. Experiments show that when entering the team, the performance of Redis is higher than RabbitMQ when the data is relatively small, and if the data size exceeds 10K, Redis is unbearably slow; when leaving the team, regardless of the size of the data, Redis shows very good performance , And RabbitMQ's dequeue performance is much lower than Redis.
7.5kafka
A sub-project under Apache, a high-performance distributed Publish/Subscribe message queue system implemented using scala, has the following characteristics:
Fast persistence: Through the disk sequential read and write and zero copy mechanism, message persistence can be performed under O(1) system overhead;
high throughput: a throughput rate of 10W/s can be reached on an ordinary server;
high Accumulation: Supports consumers offline for a long time under topic, and a large amount of message accumulation;
Fully distributed system: Broker, Producer, and Consumer all natively automatically support distributed, relying on zookeeper to automatically achieve complex balance;
Support Hadoop data parallel loading: For image Hadoop has the same log data and offline analysis system, but requires the limitation of real-time processing. This is a feasible solution.
7.6ZeroMQ
Known as the fastest message queuing system, it is specially developed for high throughput/low latency scenarios. It is often used in financial applications and focuses on real-time data communication scenarios.
However, ZeroMQ only provides non-persistent queues. If the machine is down, data will be lost.
Features:
Lock-free queue model: For the data exchange channel pipe between cross-thread interactions (user and session), use lock-free Queue algorithm CAS; Asynchronous events are registered at both ends of the pipe. When reading or writing messages to the pipe, the read and write events are automatically triggered.
Batch processing algorithm: For batch messages, adaptive optimization is carried out, and messages can be received and sent in batches.
Thread binding under multi-core, no CPU switching: different from the traditional multi-threaded concurrency mode, semaphore or critical section, zeroMQ makes full use of the advantages of multi-core, each core is bound to run a worker thread to avoid multi-threading CPU switching overhead.
2. Comparison of main message middleware
