About the use of message queues

About the use of message queues

1. Overview of
message queues Message queue middleware is an important component in distributed systems. It mainly solves problems such as application decoupling, asynchronous messages, and traffic cutting, and realizes a high-performance, high-availability, scalable and eventually consistent architecture. Currently, the most used message queues are ActiveMQ, RabbitMQ, ZeroMQ, Kafka, MetaMQ, RocketMQ

2. Application Scenarios
of Message Queuing The following describes the commonly used usage scenarios of message queues in practical applications. Four scenarios of asynchronous processing, application decoupling, traffic cutting and message communication.

2.1 Asynchronous processing
Scenario description: After the user registers, he needs to send a registration email and a registration SMS. There are two traditional methods: 1. Serial method; 2. Parallel method
a. Serial method: After the registration information is successfully written into the database, send the registration email, and then send the registration SMS. After the above three tasks are all completed, return to the client.

b. Parallel mode: After the registration information is successfully written into the database, the registration message is sent at the same time as the registration email is sent. After the above three tasks are completed, return to the client. The difference from serial is that the parallel method can improve the processing time

Assuming that each of the three business nodes uses 50 milliseconds, without considering other overheads such as the network, the serial time is 150 milliseconds, and the parallel time may be 100 milliseconds.
Because the number of requests processed by the CPU per unit time is fixed, it is assumed that the CPU throughput is 100 times per second. Then the number of requests that the CPU can process in 1 second in serial mode is 7 times (1000/150). The number of requests processed in parallel is 10 times (1000/100).
Summary: As described in the above case, the performance (concurrency, throughput, response time) of the traditional system will have bottlenecks. How to solve this problem?

The introduction of message queue will not be necessary for business logic, asynchronous processing. The reconstructed architecture is as follows:

According to the above agreement, the user's response time is equivalent to the time when the registration information is written into the database, which is 50 milliseconds. After registering an email, sending a short message and writing it to the message queue, it returns directly, so the speed of writing to the message queue is very fast and can be basically ignored, so the user's response time may be 50 milliseconds. So after the architecture change, the throughput of the system increased to 20 QPS per second. It is 3 times better than serial and twice better than parallel.

2.2 Application decoupling
Scenario description: After the user places an order, the order system needs to notify the inventory system. Traditionally, the order system calls the interface of the inventory system. As shown below:

Disadvantages of the traditional model: If the inventory system cannot be accessed, the order to reduce inventory will fail, resulting in order failure, and the order system is coupled with the inventory system

How to solve the above problems? The solution after introducing the application message queue, as shown in the figure below:

Order system: After the user places an order, the order system completes the persistent processing, writes the message to the message queue, and returns the user's order successfully placed
. /Push method to obtain order information, and the inventory system performs inventory operations according to the order information.
Suppose: the inventory system cannot be used normally when placing an order. It does not affect the normal order placement, because after the order is placed, the order system writes to the message queue and no longer cares about other subsequent operations. Realize application decoupling of order system and inventory system

2.3 Traffic clipping
Traffic clipping is also a common scenario in message queues, and is generally used in spike or group grab activities.
Application scenario: spike activity, generally due to excessive traffic, the traffic surges and the application hangs. To solve this problem, it is generally necessary to join a message queue at the front end of the application.
a. The number of active people can be controlled.
b. It can alleviate

the request of high-traffic overwhelm the application user in a short time. After the server receives it, it will first write to the message queue. If the message queue length exceeds the maximum number, the user request will be discarded directly or the error page will be jumped.
The seckill business performs follow-up processing according to the request information in the message queue

2.4 Log processing
Log processing refers to the use of message queues in log processing, such as the application of Kafka, to solve the problem of a large number of log transmissions. The architecture is simplified as follows

Log collection client, responsible for log data collection, regularly written and written to Kafka queue
Kafka message queue, responsible for log data reception, storage and forwarding
Log processing application: subscribe and consume  log data in Kafka queue

2.5 Message communication
Message communication means that message queues generally have built-in efficient communication mechanisms, so they can also be used in pure message communication. For example, implementing point-to-point message queues, or chat rooms, etc.
Point-to-point communication:

Client A and Client B use the same queue for message communication.

Chat room communication:

Client A, Client B, and Client N subscribe to the same topic to publish and receive messages. Achieving a chat room-like effect.

The above are actually two message modes of message queues, point-to-point or publish-subscribe mode. The model is a schematic diagram for reference.

3. Example of message middleware 
3.1 E-commerce system

The message queue adopts high-availability and persistent message middleware. Such as Active MQ, Rabbit MQ, Rocket MQ.
(1) After the application completes the main logic processing, it writes to the message queue. Whether the message is sent successfully can enable the confirmation mode of the message. (After the message queue returns the status of receiving the message successfully, the application will return to ensure the integrity of the message)
(2) Extend the process (send text messages, delivery processing) to subscribe to queue messages. Use push or pull to get messages and process them.
(3) While the message decouples the application, it brings about the problem of data consistency, which can be solved by the final consistency method. For example, the main data is written into the database, and the extended application realizes the follow-up processing based on the message queue in combination with the database method according to the message queue.

3.2 The log collection system

is divided into four parts: the Zookeeper registry, the log collection client, the Kafka cluster and the Storm cluster (OtherApp).
Zookeeper registry, which proposes load balancing and address lookup services
Log collection client, used to collect application system logs and push data to kafka queue
Kafka cluster: receive, route, store, forward and other message processing
Storm cluster: in the At the same level, the data in the queue is consumed by pulling

Fourth, JMS message service
Speaking of message queue, we have to mention JMS. JMS (JAVA Message Service, java message service) API is a standard/specification of a message service that allows application components to create, send, receive and read messages based on the JavaEE platform. It makes distributed communication less coupled, messaging services more reliable and asynchronous.
In the EJB architecture, there are message beans that can be seamlessly integrated with the JM message service. In the J2EE architecture pattern, there is the message server pattern, which is used to realize the direct decoupling of messages and applications.

4.1 Message Model
In the JMS standard, there are two message models P2P (Point to Point), Publish/Subscribe (Pub/Sub).

4.1.1 P2P mode

P2P mode includes three roles: message queue (Queue), sender (Sender), receiver (Receiver). Each message is sent to a specific queue, and the receiver gets the message from the queue. Queues hold messages until they are consumed or time out.

Features of P2P There is
only one consumer (Consumer) per message (that is, once consumed, the message is no longer in the message queue)
There is no time dependency between the sender and the receiver, that is, when the sender sends After the message, whether the receiver is running or not, it does not affect the message being sent to the queue.
The receiver needs to reply to the queue after successfully receiving 
the message. If every message you want to send will be processed successfully, then you need P2P mode .

4.1.2 Pub/Sub Mode

Contains three roles Topic, Publisher, Subscriber Multiple publishers send messages to Topic, and the system delivers these messages to multiple subscribers.

Features of Pub/Sub
Each message can have multiple consumers
There is a time dependency between publishers and subscribers. For a subscriber of a topic (Topic), it must create a subscriber before it can consume the message of the publisher.
In order to consume the message, the subscriber must keep the running state.
In order to alleviate such strict time correlation, JMS allows the subscriber to create A durable subscription. This way, even if the subscriber is not activated (running), it can receive messages from the publisher.
If the message you want to send can be processed without any processing, or only processed by one messager, or can be processed by multiple consumers, then the Pub/Sub model can be used.

4.2 Message consumption
In JMS, the generation and consumption of messages are asynchronous. For consumption, JMS messagers can consume messages in two ways.
(1) Synchronization
Subscribers or receivers receive messages through the receive method, and the receive method will block until the message is received (or before the timeout).

(2) Asynchronous
Subscriber or receiver can be registered as a message listener. When the message arrives, the system automatically calls the listener's onMessage method.

JNDI: Java Naming and Directory Interface, is a standard Java naming system interface. Services can be found and accessed on the web. By specifying a resource name, the name corresponds to a record in the database or naming service, and returns the information necessary to establish a resource connection.
JNDI plays a role in finding and accessing the destination or source of the message in JMS.

Five, commonly used message queue

General commercial containers, such as WebLogic and JBoss, all support the JMS standard, which is very convenient for development. But free ones such as Tomcat, Jetty, etc. need to use third-party message middleware. This part introduces the commonly used message middleware (Active MQ, Rabbit MQ, Zero MQ, Kafka) and their characteristics.

5.1 ActiveMQ
ActiveMQ is the most popular and powerful open source message bus produced by Apache. ActiveMQ is a JMS Provider implementation that fully supports the JMS1.1 and J2EE 1.4 specifications. Although the JMS specification has been published for a long time, JMS still plays a special role in today's J2EE applications.

ActiveMQ features are as follows:
1. Write clients in multiple languages ​​and protocols. Languages: Java, C, C++, C#, Ruby, Perl, Python, PHP. Application protocols: OpenWire, Stomp REST, WS Notification, XMPP, AMQP
⒉ Fully supports JMS1.1 and J2EE 1.4 specifications (persistence, XA messages, transactions)
⒊ Support for Spring, ActiveMQ can be easily embedded into systems using Spring It also supports the features of Spring 2.0.
⒋ Passed the test of common J2EE servers (such as Geronimo, JBoss 4, GlassFish, WebLogic). Through the configuration of JCA 1.5 resource adaptors, ActiveMQ can be automatically deployed to any compatible J2EE 1.4 commercial server
⒌ Supports multiple transmission protocols: in-VM, TCP, SSL, NIO, UDP, JGroups, JXTA
⒍ Supports high-speed message persistence through JDBC and journal
⒎ The design ensures high-performance clusters, Client-server, point-to-point
⒏ Support Ajax
⒐ Support integration with Axis
⒑ It is easy to call the embedded JMS provider for testing

5.2 Kafka
Kafka is a high-throughput distributed publish-subscribe messaging system that can process all action streaming data in consumer-scale websites. Such actions (web browsing, searching, and other user actions) are a key factor in many social functions on the modern web. This data is usually addressed by processing logs and log aggregation due to throughput requirements. This is a viable solution for log data and offline analysis systems like Hadoop, but with the constraints of real-time processing. The purpose of Kafka is to unify online and offline message processing through Hadoop's parallel loading mechanism, and to provide real-time consumption through cluster machines.

Kafka is a high-throughput distributed publish-subscribe messaging system with the following characteristics:
It provides message persistence through an O(1) disk data structure, which can maintain a long time for even terabytes of message storage. stable performance. (Data is written by appending to files, and expired data is deleted periodically) High throughput: even very common hardware Kafka can support
millions of messages per second Data Loading Kafka Related Concepts A Broker Kafka cluster contains one or more servers, which are called brokers [5] Topic Each message published to the Kafka cluster has a category, which is called a Topic. (Physically, messages of different topics are stored separately. Logically, although messages of a topic are stored on one or more brokers, users only need to specify the topic of the message to produce or consume data without worrying about where the data is stored) Partition Partition It is a physical concept. Each Topic contains one or more Partitions. Producer is responsible for publishing messages to Kafka broker Consumer message consumers and clients that read messages from Kafka broker. Consumer Group Each Consumer belongs to a specific Consumer Group (group name can be specified for each Consumer, if no group name is specified, it belongs to the default group).















It is generally used in big data log processing or in scenarios with slightly lower requirements for real-time (a small amount of delay) and reliability (a small amount of data loss).