MQ, message queue, stands for message queue . Through a typical producer and consumer model, the producer continuously produces messages to the message queue, and the consumer continuously obtains messages from the queue. Because the production and consumption of messages are asynchronous, and they only care about the sending and receiving of messages, there is no intrusion of business logic, and the decoupling between systems is easily achieved. Also known as message middleware , it conducts platform-independent data exchange through the use of an efficient and reliable message transfer mechanism, and integrates distributed systems based on data communication.

1.2 Why use MQ?

1.2.1 What are the disadvantages of traditional http requests?

1) The http request is based on the request-response model. In the case of high concurrency, the client sends a large number of requests to the server, which may lead to the accumulation of server requests. In the case of too many requests, it may cause the server to crash.

Generally, current limiting and integrated services are implemented at the entrance of the server to protect the framework.

2) When the processing logic of the http request is time-consuming, it may cause the client's response to time out. Therefore, the time-consuming business logic is handled by multi-threading or MQ as much as possible.

1.2.2 Multithreaded asynchronous task processing

When a new user applies for registration on an e-commerce platform, the client sends a request to the server, and the server performs a series of operations:

Insert member information, it takes 1s
It takes 3s to send login SMS reminder
It takes 3s to send new coupons

It takes a total of 7s for the server to respond to the client's request. During this 7s, the client is blocked, which affects the user experience.

If you use multiple threads for asynchronous processing:

After inserting the member data into the database, a separate thread is started to send login messages and newcomer coupons.

It only takes 1s for the server to respond to the client's request.

But the disadvantage of doing this is: starting the thread alone will consume the server's CPU resources, and the server will be under great pressure when processing a large number of requests.

1.2.3 MQ asynchronous task processing

If you use MQ for asynchronous processing:

First insert the member data into the database, and then deliver the message to MQ. After the MQ server pushes the message to the consumer of the message, the consumer sends a login message and newcomer coupon. [Asynchronous, decoupling]

It only takes 1s for the server to respond to the client request and does not consume CPU resources.

How does MQ realize anti-high concurrency?

MQ consumers pull messages from the MQ server for processing according to their own capabilities.

This ability to do so effectively improves the ability of message middleware to resist high concurrency, but the disadvantage of doing so is that it may cause message processing delays. 【Peak Cutting】

Ways to improve the processing speed of MQ consumer tasks:

Build consumer clusters to increase processing speed
Set up consumers to get messages in batches (by default, consumers get one message at a time)

1.3 MQ core concepts

Message: Message, data transmitted between applications, the more commonly used data format is JSON.

Broker: The MQ server side provides functions such as receiving, storing, and pulling messages.

Producer: Producer, delivers messages to Broker.

Consumer: Consumers, get messages from Broker, and process business logic.

Topic: Topic. After the Broker receives a message on a topic, it will push the message to consumers who have subscribed to the topic.

JSON (JavaScript Object Notation), JS object notation. It is a lightweight data exchange format that uses a text format completely independent of the programming language to store and represent data. The concise and clear hierarchical structure makes JSON an ideal data exchange language. It is easy for people to read and write, but also easy for machine to parse and generate, and effectively improve the efficiency of network transmission.

Example: (Online json formatting verification: https://www.bejson.com/ )
{
	"group1": [{
		"name": "LiMing",
		"gender": "male",
		"age": "9"
	}, {
		"name": "Janny",
		"gender": "female",
		"age": "8",
		"nationality": "Canada"
	}, {
		"name": "Danny",
		"gender": "male",
		"age": "10",
		"species": "dinosaur"
	}],
	"group2": [{
		"name": "Kangkang",
		"gender": "male",
		"age": "14"
	}, {
		"name": "Michael",
		"gender": "male",
		"age": "14",
		"nationality": "America"
	}, {
		"name": "Jane",
		"gender": "female",
		"age": "14",
		"nationality": "Australia"
	}, {
		"name": "Maria",
		"gender": "female",
		"age": "15",
		"nationality": "Cuba"
	}],
	"group3": {
		"name": "LiHua",
		"gender": "female",
		"age": "18",
		"remarks": "eternalGod"
	}
}

1.4 Simple implementation of MQ

1.4.1 Maven dependency

        <!-- https://mvnrepository.com/artifact/com.alibaba/fastjson -->
        <dependency>
            <groupId>com.alibaba</groupId>
            <artifactId>fastjson</artifactId>
            <version>1.2.62</version>
        </dependency>

1.4.2 Multithreading to achieve local MQ

Ideas:

Create a blocking queue as Broker; create two threads, one as Producer and the other as Consumer.

The Producer sends a message to the Broker, and the Consumer consumes the message.

Code demo:

import com.alibaba.fastjson.JSONObject;

import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.TimeUnit;

public class MyMQ {
    /**
     * Broker
     */
    private static LinkedBlockingQueue<JSONObject> broker = new LinkedBlockingQueue<>();

    public static void main(String[] args) {
        /**
         * Producer
         */
        Thread producer = new Thread(() -> {
            while (true) {
                JSONObject data1 = new JSONObject();
                JSONObject data2 = new JSONObject();
                JSONObject data3 = new JSONObject();
                JSONObject data4 = new JSONObject();
                data1.put("name", "Kangkang");
                data1.put("gender", "male");
                data1.put("age", 14);
                data2.put("name", "Michael");
                data2.put("gender", "male");
                data2.put("age", 14);
                data3.put("name", "Jane");
                data3.put("gender", "female");
                data3.put("age", 14);
                data4.put("name", "Maria");
                data4.put("gender", "female");
                data4.put("age", 15);

                JSONObject data = new JSONObject();
                Object[] array = {data1, data2, data3, data4};
                data.put("group2", array);
                broker.offer(data);
                try {
                    TimeUnit.SECONDS.sleep(5);
                } catch (InterruptedException e) {
                    e.printStackTrace();
                }
            }
        }, "生产者");
        producer.start();

        /**
         * Consumer
         */
        Thread consumer = new Thread(() -> {
            while (true) {
                JSONObject data = broker.poll();
                if (data != null) {
                    System.out.println(Thread.currentThread().getName() + "获取到数据：" + data.toJSONString());
                }
            }
        }, "消费者");
        consumer.start();
    }
}

Running result: (output once every 5s)

消费者获取到数据：{"group2":[{"gender":"male","name":"Kangkang","age":14},{"gender":"male","name":"Michael","age":14},{"gender":"female","name":"Jane","age":14},{"gender":"female","name":"Maria","age":15}]}

1.5 Current mainstream MQ

1.5.1 ActiveMQ

ActiveMQ is a product of Apache. It is a message middleware that fully supports JMS (Java Message Service) specifications. It provides a rich API, supports a variety of clients to call, and also provides a variety of cluster architecture models. The most popular and powerful open source message bus in the industry.

However, in the development process of ActiveMQ in recent years, its performance has been criticized-its throughput is not high, so large Internet companies generally will not use it, and it is more popular among small and medium enterprises.

1.5.2 Kafka

Kafka currently belongs to the top Apache project. Kafka pursues high throughput, and its initial purpose is to collect and transmit logs.

Although Kafka's performance is particularly high, it does not support transactions and does not have strict requirements for message duplication, loss, and errors.

1.5.3 RocketMQ

RocketMQ is Alibaba's open source message middleware. It is developed in pure Java and has the characteristics of high throughput, high availability, and suitable for large-scale distributed system applications. The RocketMQ idea originated from Kafka, but it is not a copy of Kafka. It optimizes the reliable transmission and transactionality of messages. It is currently widely used in Alibaba Group for transactions, recharge, stream computing, message push, log streaming, Scenarios such as binglog distribution.

1.5.4 RabbitMQ

RabbitMQ is an open source message queue developed using Erlang language and implemented based on the AMQP protocol. The main characteristics of AMQP are message-oriented, queue, routing (including point-to-point and publish/subscribe), reliability, and security. The AMQP protocol is more used in scenarios where data consistency, stability, and reliability are very high in enterprise systems, and the requirements for performance and throughput are second.

[I write here temporarily, if I have time in the future to write about RabbitMQ and Kafka]

加油！ (d • _ •) d

Fight against Java multithreading and high concurrency-MQ articles

1 SQM

1.1 What is MQ?