Preface
RabbitMQ, as one of the current mainstream message middlewares, is undoubtedly an important part of the growth path of our Java back-end development technology. In this article, Lizhi will sort out the knowledge of getting started with RabbitMQ. The article involves the basic concepts of RabbitMQ and its environment configuration. , Lizhi’s RabbitMQ is deployed on Docker. Friends who need it can directly read Section 2.3.3~~~ Lizhi also sorted out the simplest two of the six core modes of RabbitMQ: simple mode and working mode, and the corresponding Message response mechanism knowledge, I hope my summary can be useful to friends who are learning!
Article directory
2.3.1 Enter RabbitsMQ official website to download
2.3.2 Copy local files to Docker container
2.3.3 Container installation RabbitMQ
2.3.4 Install RabbitMQ directly with Docker
3. Simple mode and working mode
3.3 Message response mechanism
3.3.2 Other response mechanisms
3.3.4 Messages are automatically re-queued
3.3.5 Code example of message manual response mechanism
1. Related concepts of MQ
1.1 MQ
MQ (message queue) is essentially a queue, which follows the first-in-first-out principle. However, the content stored in the queue is only messages. It is also a cross-process communication mechanism used to transmit messages upstream and downstream. In the Internet architecture, MQ is a very common upstream and downstream "logical decoupling + physical decoupling" message communication service. After using MQ, the upstream message sending only needs to rely on MQ and does not need to rely on other services.
1.2MQ characteristics
Traffic peak reduction
Due to the unique structure of MQ, access requests need to be queued and waited. When a large number of requests reach our server system, in order to prevent downtime, we can use MQ to reduce traffic peaks, but this will also bring about a decrease in performance. .
Application decoupling
If there is a system with a master-slave structure, if a queue is not used for direct coupling, failure of any subsystem will cause abnormal ordering operations. However, after using the message queue method, even if a subsystem function fails, the user will not feel that the system has failed, because the system request operation instructions are stored in the message queue and can be processed again when the system recovers.
Asynchronous processing
For example, A wants to call an API to enable B to perform time-consuming business such as big data query. From the perspective of resource utilization, we naturally cannot let A wait for B's response. B will no longer directly return the message, but will send the message after the processing is completed. Put it into an MQ, and then send it to A from MQ. In this way, A can obtain asynchronously processed messages.
1.3 Several different MQs
kafka
Kafka is a message middleware specifically designed for big data. It has become famous for its million-level TPS throughput and has quickly become the darling of the big data field. It plays a decisive role in the process of data collection, transmission, and storage.
Advantages: Excellent performance, single-machine write TPS is about one million items/second. The biggest advantage is high throughput. The timeliness and MS-level availability are very high. Kafka is distributed. There are multiple copies of one data. Even if a few machines are down, there will be no loss of data or unavailability. Consumers use the Pu‖ method to obtain messages, and the messages are in order. Control ensures that all messages are consumed and only consumed once. The function is relatively simple, mainly supporting simple MQ functions. It is used on a large scale for real-time computing and log collection in the field of big data. Disadvantages
: Kafka has more than 64 queues/partitions on a single machine, and the load will obviously increase. The more queues, the higher the load. The higher the value, the longer the response time for sending
messages . When using the short polling method, the real-time performance depends on the polling interval. Retry is not supported for consumption failures. Message order is supported, but when an agent goes down, message chaos will occur. order, community updates are slow;
RocketMQ
RocketMQ is an open source product from Alibaba and is implemented in Java language. It referred to Kafka during design and made some
improvements of its own. It is widely used by Alibaba in order, transaction, recharge, stream computing, message push, log streaming processing, binglog distribution and other scenarios.
Advantages: Single machine throughput of 100,000 levels, very high availability, distributed architecture, zero message loss, MQ function is relatively complete, distributed, good scalability, supports 1 billion level message accumulation, and will not cause problems due to accumulation The performance is reduced. The source code is jva. We can read the source code ourselves and customize our own company's MQ.
Disadvantages: There are not many supported client languages, currently jva and c++, of which c++ is immature. The community activity is average and it is not
implemented in the MQ core. JMS and other interfaces, some systems need to modify a lot of code to migrate
RabbitMQ
Released in 2007, it is a reusable enterprise messaging system based on AMQP (Advanced Message Queuing Protocol). It is one of the most mainstream messaging middleware currently.
Advantages: Due to the high concurrency characteristics of the erlang language, the performance is better; the throughput reaches 10,000 levels, the MQ function is relatively complete, robust, stable, easy to use, cross-platform, and supports multiple languages such as: Python, .Ruby, .NET, Java , JMS, C, PHP, ActionScript, .XMPP, STOMP
2. Introduction to RabbitMQ
RabbitMQ is a message middleware that does not process messages but receives, stores and forwards message data.
2.1 Four basic concepts
Producer: a program that generates data and sends messages
Switch: It is a very important component of RabbitMQ. On the one hand, it receives messages from producers, and on the other hand, it pushes messages to the queue. The switch must know exactly how to process the messages it receives, whether to push these messages to a specific queue or to multiple queues, or to discard the messages. This depends on the type of switch.
Queue: It is a data structure used internally by RabbitMQ. Although messages flow through RabbitMQ and the application, they can only be stored in the queue. A queue is limited only by the memory and disk limitations of the host and is essentially a large message buffer. Many producers can send messages to a queue, and many consumers can try to receive data from a queue. This is how we use queues
Consumer: Consumption and reception have similar meanings. A consumer is mostly a program waiting to receive messages. Please note that the producer, consumer and message middleware are often not on the same machine. The same application can be both a producer and a consumer.
2.2 RabbitMQ structure
The following figure briefly depicts the structure and operating mechanism of RabbitMQ:
- Broker: An application that receives and distributes messages. RabbitMQ Server is the Message Broker message entity.
- Virtual host: Designed for multi-tenancy and security reasons, the basic components of AMQP are divided into a virtual group, similar to the namespace concept in the network. When multiple different users use the services provided by the same RabbitMQ server, multiple vhosts can be divided, and each user creates exchange/queue, etc. in their own vhost.
- Connection: channel, TCP connection between publisher/consumer and broker
- Channel: If a Connection is established every time RabbitMQ is accessed, the overhead of establishing a TCP Connection will be huge and the efficiency will be low when the message volume is large. Channel is a logical connection established inside the connection . If the application supports multi-threading, each thread usually creates a separate channel for communication. The AMQP method includes the channel id to help the client and message broker identify the channel, so the channels are completely isolated . of. Channel, as a lightweight Connection, greatly reduces the operating system's overhead in establishing TCP connections.
- Exchange: The message reaches the first stop of the broker. According to the distribution rules, it matches the routing key in the query table and distributes the message to the queue. Commonly used types are: direct (point-to-point)), topic (publish-subscribe) and fanout (multicast)
- Exchange: switch
- Queue: Queue
- Producer: producer
- Consumer: consumer
2.3 RabbitMQ installation
In view of the fact that most of the other methods are relatively comprehensive on the Internet, Lizhi gives a different installation method here. Because Lizhi's own cloud server has expired, it uses Docker to virtualize a Centos system on the computer as its own Linux server. Here, Litchi will just outline his own approach:
first, after installing Docker and Docker Desktop, he will start to pull the Centos image and run the container. You may encounter many problems here like Litchi, but don’t be afraid to follow the tutorial step by step:
https://blog.csdn.net/GoodburghCottage/article/details/131413312
If you encounter an Exited (255) error, just restart the container.
After pulling the Centos image, it looks like this.
2.3.1 Enter RabbitsMQ official website to download
You can decide on the version here.
erlang: Release 22.2.1 · rabbitmq/erlang-rpm · GitHub
RabbitMQ 下载： Release RabbitMQ 3.8.11 · rabbitmq/rabbitmq-server · GitHub
Download in the specified directory, please remember your path name
2.3.2 Copy local files to Docker container
In the command line, cd to enter the path where the current file is stored and use the docker cp command to transfer the file to Docker.
The syntax for executing docker cp is:
docker cp moves the file to the name of the current directory container name | container id: the path where the file is stored in the container
Check whether the file is successfully copied to the specified directory
You can see that it has been successful!
2.3.3 Container installation RabbitMQ
yum install -y socat
rpm -ivh erlang-22.2.1-1.el7.x86_64.rpm
rpm -ivh rabbitmq-server-3.8.11-1.el7.noarch.rpmAt this step you may find that an error is reported:
Failed to get D-Bus connection: No such file or directory
I have been checking this bug for a long time and I still don’t know how to solve it. You guys can teach me in the comment area~~~
Then Litchi changed his approach:
2.3.4 Install RabbitMQ directly with Docker
#使用docker拉取rabbitmq镜像
docker pull rabbitmq:management
#创建一个数据卷用来持久化rabbitmq中的数据
docker volume create rabbitmq-home
#创建rabbitmq的docker容器并设置rabbitmq登录管理后台的用户名和密码
docker run -id --name=rabbitmq -v rabbitmq-home:/var/lib/rabbitmq -p 15672:15672 -p 5672:5672 -e RABBITMQ_DEFAULT_USER=root -e RABBITMQ_DEFAULT_PASS=123456 rabbitmq:managementIt should be noted that the access port of rabbitmq management background here is 15672. When we access, we only need to add our corresponding server IP
The page after entering looks like this.
One step to reach the destination hahahaha~~~
2.4 Dependency introduction
Before learning the six core modes of RabbitMQ, we need to configure the environmental dependencies of RabbitMq in the project and configure it in the pom.xml file:
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>8</source>
<target>8</target>
</configuration>
</plugin>
</plugins>
</build>
<!--rabbitmq依赖客户端-->
<dependencies>
<dependency>
<groupId>com.rabbitmq</groupId>
<artifactId>amqp-client</artifactId>
<version>5.8.0</version>
</dependency>
<!--操作文件流的一个依赖-->
<dependency>
<groupId>commons-io</groupId>
<artifactId>commons-io</artifactId>
<version>2.6</version>
</dependency>
</dependencies>Update maven to complete the dependency introduction!
3. Simple mode and working mode
Six core mode classifications : simple mode, working mode, publish and subscribe mode, routing mode, topic mode, publish and confirm mode
3.1 Simple queue mode
producer code
For the use of RabbitMQ, we need to create a connection factory object based on ConnectionFactory and set the corresponding connection account and factory IP. After creating the connection and channel, use queueDeclare to declare the queue you want to create.
Five parameters of queueDeclare method:
- First parameter: Queue name
- Second parameter: Set whether the messages in the queue are persistent. If not enabled by default, the messages will be stored in memory.
- The third parameter: Set whether the queue is only for one consumer to consume and whether to share messages. Set it to true to set multiple consumers to consume.
- The fourth parameter: whether to automatically delete the queue after the last consumer disconnects, true to automatically delete, false not to automatically delete
- The fifth parameter: other parameters
After declaring the queue, you need to use basicPublish() to send the message body.
Specific parameter meaning:
- Switch name
- Queue name (key value of route)
- Other parameter messages
- The message body of the sent message
package com.crj.rabbitmq;
import com.rabbitmq.client.*;
public class Producer {
//定义队列名
public static final String QUEUE_NAME = "hello";
//发送消息
public static void main(String[] args) throws Exception {
//创建连接工厂
ConnectionFactory factory = new ConnectionFactory();
//工厂IP连接RabbitMQ的队列
factory.setHost("127.0.0.1");
factory.setUsername("root");
factory.setPassword("123456");
//创建连接
Connection connection = factory.newConnection();
//创建一个信道
Channel channel = connection.createChannel();
//生成一个队列
/**
* 几个属性：
* 1.队列名称
* 2.队列里面的消息是否持久化，默认不开启的话将消息存储在内存中
* 3.第三个参数设置队列是否只供一个消费者进行消费 是否进行消息共享，设置为true设置多个消费者消费
* 4.最后一个消费者端断开连接之后是否自动删除队列，true自动删除，false不自动删除
* 5.其它参数
*/
channel.queueDeclare(QUEUE_NAME,false,false,false,null);
//定义消息
String message = "hello word";
channel.basicPublish("",QUEUE_NAME,null,message.getBytes());
System.out.println("消息已发送");
}
}
consumer code
Consumers, like producers, need to set up RabbitMQ queue information and create connections and channels. Finally, basicConsume is used to consume the message. The basicConsume() method requires four parameters to be passed in. The specific meaning is:
- Which queue to consume
- Whether to automatically respond after successful consumption , true means automatic response, false means manual response
- Callback function for unsuccessful consumption
- Callback function for consumer to cancel consumption
For the callback function, we need to implement two interfaces, DeliverCallback and CancelCallback, and use lambda expressions instead of internal implementation classes.
package com.crj.rabbitmq;
import com.rabbitmq.client.*;
public class Consumer {
public static final String QUEUE_NAME = "hello";
public static void main(String[] args) throws Exception {
ConnectionFactory factory = new ConnectionFactory();
//工厂IP连接RabbitMQ的队列
factory.setHost("127.0.0.1");
factory.setUsername("root");
factory.setPassword("123456");
//创建连接
Connection connection = factory.newConnection();
//创建一个信道
Channel channel = connection.createChannel();
//使用Lambda表达式来定义消费和取消消费的回调函数
//消费回调函数
DeliverCallback deliverCallback = (consumerTag, message)->{
System.out.println("message is "+new String(message.getBody()));
};
//取消消费的回调函数
CancelCallback cancelCallback = (consumerTag)->{
System.out.println(consumerTag+"取消消息消费");
};
//消费消息
/**
* 消费哪个队列
* 消费成功之后是否自动应答，true为自动应答，false为手动应答
* 未成功消费的回调函数
* 消费者取消消费的回调函数
*/
channel.basicConsume(QUEUE_NAME,true,deliverCallback,cancelCallback);
}
}
3.2 Working mode
When a producer sends a large number of messages into the message queue, the consumer has only one worker thread, which obviously cannot meet the demand and may even cause the queue to be blocked. Therefore, a mode of using multiple worker threads to consume messages is proposed, which is the working mode. A simple understanding is that it is an idea of the big sponsor Party A. All of you in a department need to work together to complete the tasks given by Party A. What needs to be noted here is that the message can only be processed once, so RabbitMQ uses a polling message distribution mechanism .
Extraction channel creation tool class
package com.crj.rabbitmq.utils;
import com.rabbitmq.client.Channel;
import com.rabbitmq.client.Connection;
import com.rabbitmq.client.ConnectionFactory;
public class RabbitMqUtil {
//抽象一个静态类来定义一个获取信道的方法
public static Channel getChannel() throws Exception {
//创建连接工厂
ConnectionFactory factory = new ConnectionFactory();
//工厂IP连接RabbitMQ的队列
factory.setHost("127.0.0.1");
factory.setUsername("root");
factory.setPassword("123456");
//创建连接
Connection connection = factory.newConnection();
//创建一个信道
Channel channel = connection.createChannel();
return channel;
}
}
Create a publisher and multiple consumers respectively to verify this model. Only the publisher's demo is given here. The consumer's demo is consistent with the above.
package com.crj.rabbitmq.simple;
import com.rabbitmq.client.*;
public class Producer {
//定义队列名
public static final String QUEUE_NAME = "hello";
//发送消息
public static void main(String[] args) throws Exception {
//创建连接工厂
ConnectionFactory factory = new ConnectionFactory();
//工厂IP连接RabbitMQ的队列
factory.setHost("127.0.0.1");
factory.setUsername("root");
factory.setPassword("123456");
//创建连接
Connection connection = factory.newConnection();
//创建一个信道
Channel channel = connection.createChannel();
//生成一个队列
channel.queueDeclare(QUEUE_NAME,false,false,false,null);
//定义消息
String message = "hello word";
//交换机名 队列名|路由的key值 其它参数消息 发送消息的消息体
channel.basicPublish("",QUEUE_NAME,null,message.getBytes());
System.out.println("消息已发送");
}
}
3.3 Message response mechanism
As mentioned earlier in the working mode, a message in the queue can only be processed once by the consumer, but when will the queue delete the message? Is it possible that a consumer will crash during the process of processing the message, which will occur? The message is lost. To avoid this situation, RabbitMQ uses a message response mechanism. This mechanism ensures that the message queue will not delete the message until the consumer has completely processed the message .
3.3.1 Automatic answer
It must be used under certain conditions: that is, the message is deemed to have been successfully delivered immediately after it is sent. This model requires a trade-off between high throughput and data transmission security, and it is easy for messages to be lost if the consumer is down or there is a connection problem . Therefore, this mode is only suitable when it is ensured that the consumer can be efficiently concurrent and can carry the current message throughput .
3.3.2 Other response mechanisms
Compared with automatic answering, the advantage of manual answering is that it can respond in batches and reduce network congestion.
Channel.basicAck: Positive response confirmation, RabbitMQ will only discard the message body in the queue after it has obtained the message that the message has been successfully processed.
Channel.basicNack: used for negative confirmation, no confirmation required, the message will be discarded after it is claimed.
Channel.basicReject: Refuse to confirm. Compared with basicNack, it has one more parameter for batch processing of messages.
3.3.3 Batch response Multiple
Batch deletion is actually to decide whether to send all the messages in the queue at one time. Here, a boolean parameter is generally used to select. True represents a batch response, and false represents only responding to the information at the head of the current queue.
Note : The response message is the message that has been processed, that is, the message that the consumer has consumed.
It should also be noted that we need to use batch processing with caution to avoid direct response to messages in the queue that have not yet received ACK, resulting in message loss.
3.3.4 Messages are automatically re-queued
In the message response mechanism, if the message queue RabbitMQ does not receive an Ack from the consumer (I won’t go into details here), the message will be reassigned to a new consumer for consumption. This is the re-enqueue of the message.
3.3.5 Code example of message manual response mechanism
To manually respond, you first need to set the autoAck parameter to false in channel.basicConsumer(), and then you need to set the logic of manual response yourself. There are two parameters in the channel.basicAck() method:
- The first parameter: Get the tag of the current message
- Second parameter: Set whether to respond in batches
package com.crj.rabbitmq.UnAutoReply;
import com.crj.rabbitmq.utils.RabbitMqUtil;
import com.crj.rabbitmq.utils.SleepUtils;
import com.rabbitmq.client.CancelCallback;
import com.rabbitmq.client.Channel;
import com.rabbitmq.client.DeliverCallback;
public class Work01 {
public static final String TASK_QUEUE_ACK = "ack_queue";
public static void main(String[] args) throws Exception {
Channel channel = RabbitMqUtil.getChannel();
System.out.println("我是消费者 1 ，正在等待接收消息》》》》");
System.out.println("接收消息的时间较短");
DeliverCallback deliverCallback = (consumerTag,message)->{
//睡眠时间
SleepUtils.sleep(1);
System.out.println("接收到的消息："+new String(message.getBody(),"UTF-8"));
/**
* 手动应答的逻辑
*/
//获取消息标记
long deliveryTag = message.getEnvelope().getDeliveryTag();
channel.basicAck(deliveryTag,false);
};
CancelCallback cancelCallback = (consumerTarget)->{
System.out.println(consumerTarget + "消费者取消消费接口的回调逻辑");
};
//采用手动应答
boolean autoAck = false;
channel.basicConsume(TASK_QUEUE_ACK,autoAck,deliverCallback,cancelCallback);
}
}
Simply test the manual response mechanism and you can see that the message will not disappear~
Summarize
Above, Lizhi briefly sorted out the two modes of RabbitMQ. In subsequent articles, Lizhi will also sort out the remaining modes and the knowledge of RabbitMQ clusters. Get RabbitMQ in one week, come on!
Today has become the past, but we still look forward to the future tomorrow! I am Litchi, and I will accompany you on the road of technological growth~~~
If the blog post is helpful to you, you can give Lizhi three clicks. Your support and encouragement are Lizhi’s biggest motivation!
If the content of the blog post is incorrect, you are also welcome to criticize and correct it in the comment area below! ! !