A Quick Start to RabbitMQ for Advanced Java (1)

RabbitMQ Quick Start for Advanced Java

Related concepts

What is MQ

MQ (message queue) is a message queue. We first need to know that the basic nature of the queue is first in first out, and the message queue uses the data structure of the queue to store message messages. . In the Internet architecture, MQ is a very common upstream and downstream "logical decoupling + physical decoupling" message communication service. After using MQ, the upstream of the message sending only needs to rely on MQ, and does not need to rely on other services. The so-called upstream and downstream means that the upstream is responsible for sending messages, and the downstream is responsible for receiving messages.

Understand what is MQ, so why do we use MQ, that is to say, the benefits of using MQ?

1. The first is traffic peak reduction. Take Taobao’s Double 11 as an example. We all know that there are a lot of users who want to buy goods on Taobao’s Double 11. The number of orders processed by the server is limited. If this number exceeds this number, the server will It will crash, that is, the traffic is particularly large. If you use MQ, you can store some message processing in the message queue MQ. Although there will be a delay, at least the server will not crash. This is the peak of traffic.
2. The second application scenario is application decoupling. The calls between modules are very complicated. In order to reduce the call dependencies between modules, MQ is used to separate the two systems, and the system interface is not directly called. To reduce the dependency between the two systems is application decoupling. . Isn't this what we have always said that a good project starts from "high cohesion, low coupling",
3. Asynchronous processing. For example, after the user registers, he needs to send registration emails and registration SMS. There are two traditional methods: 1. Serial method 2. Parallel method. If you use the serial method, you need to wait for the returned result before you can process the registration After the code, in parallel, it is similar to opening another thread to process the registration operation separately.

Common MQ

ActiveMQ

Advantages: stand-alone throughput of 10,000, timeliness of ms, high availability, high availability based on master-slave architecture, low probability of data loss due to message reliability

Disadvantages: The official community now maintains less and less ActiveMQ 5.x, and it is less used in high-throughput scenarios.

Kafka

The killer feature of big data, when it comes to message transmission in the field of big data, Kafka cannot be avoided. This message middleware born for big data is famous for its million-level TPS throughput and has quickly become a big data platform. The darling of the field, it plays a pivotal role in the process of data collection, transmission, and storage. It has been adopted by large companies such as LinkedIn, Uber, Twitter, Netflix, etc.

Advantages : Excellent performance, the TPS of a single computer is about one million pieces/second, and the biggest advantage is the high throughput . Timeliness and ms-level availability are very high. Kafka is distributed. There are multiple copies of one data, and a few machines are down, so no data will be lost or unavailable. Consumers use the Pull method to obtain messages, and the messages are ordered. Through the control It can ensure that all messages are consumed and only consumed once; there is an excellent third-party Kafka web management interface Kafka-Manager; it is relatively mature in the log field and is used by many companies and multiple open source projects; function support: The function is relatively simple, mainly Supports simple MQ functions, real-time computing and log collection in the field of big data are used on a large scale

Disadvantages : If Kafka has more than 64 queues/partitions on a single machine, the load will increase significantly. The more queues, the higher the load, and the longer the response time for sending messages. Using the short polling method, the real-time performance depends on the polling interval. Consumption failure does not support retry; message order is supported, but when an agent goes down, messages will be out of order, and community updates will be slow .

RocketMQ

RocketMQ is an open source product from Alibaba, implemented in the Java language, referring to Kafka during design, and making some improvements of its own. It is widely used by Alibaba in scenarios such as orders, transactions, recharges, stream computing, message push, log stream processing, and binglog distribution.

Advantages: stand-alone throughput of 100,000 levels , very high availability, distributed architecture, zero loss of messages , relatively complete MQ function, distributed, good scalability, support for 1 billion-level message accumulation , and will not be lost due to Accumulation leads to performance degradation. The source code is java. We can read the source code by ourselves and customize the MQ of our company.

Disadvantages: There are not many client languages ​​supported , currently java and c++, among which c++ is immature; the community is generally active, and interfaces such as JMS have not been implemented in the MQ core, and some systems need to modify a lot of code to migrate.

RabbitMQ

Released in 2007, it is a reusable enterprise messaging system based on AMQP (Advanced Message Queuing Protocol), and is one of the most mainstream messaging middleware at present.

Advantages: Due to the high concurrency feature of the erlang language, the performance is better; the throughput reaches 10,000 levels, and the MQ function is relatively complete, robust, stable, easy to use, cross-platform, and supports multiple languages ​​such as: Python, Ruby, .NET, Java, JMS, C, PHP, ActionScript, XMPP, STOMP, etc., support AJAX and complete documentation; the management interface provided by open source is very good, easy to use, and the community is highly active; the update frequency is quite high

Official website update: https://www.rabbitmq.com/news.html(opens new window)

Disadvantages: The commercial version needs to be charged, and the learning cost is high

RabbitMQ

RabbitMQ is a messaging middleware: it accepts and forwards messages.

You can think of it as a courier site. When you want to send a package, you put your package in the courier station, and the courier will eventually deliver your courier to the recipient. According to this logic, RabbitMQ is a courier Station, a courier will deliver the express for you.

The main difference between RabbitMQ and the express station is that it does not process express mail but receives, stores and forwards message data.

Official website: https://www.rabbitmq.com/#features(opens new window)

Four core concepts of RabbitMQ

• producer

  The program that generates the data sending message is the producer

• switch

  The switch is a very important part of RabbitMQ. On the one hand, it receives messages from producers, and on the other hand, it pushes messages to the queue. The switch must know exactly what to do with the messages it receives, whether to push them to a specific queue or to multiple queues, or to discard the messages, depending on the type of switch

• queue

  Queues are a data structure used internally by RabbitMQ, and although messages flow through RabbitMQ and applications, they can only be stored in queues. The queue is constrained only by the host's memory and disk limitations and is essentially a large message buffer. Many producers can send messages to a queue, and many consumers can try to receive data from a queue. This is how we use the queue

• consumer

  Consuming has a similar meaning to receiving. Most of the time, a consumer is a program waiting to receive messages. Please note that producers, consumers and message middleware are often not on the same machine. The same application can be both a producer and a consumer.

Introduction to various terms in RabbitMQ

• Broker

  The application for receiving and distributing messages, RabbitMQ Server is Message Broker

• Virtual host

  Designed for multi-tenancy and security factors, the basic components of AMQP are divided into a virtual group, similar to the namespace concept in the network. When multiple different users use the services provided by the same RabbitMQ server, multiple vhosts can be divided, and each user creates an exchange/queue in his own vhost, etc.

• Connection

  TCP connection between publisher/consumer and broker

• Channel

  If a Connection is established every time RabbitMQ is accessed, the overhead of establishing a TCP Connection when the message volume is large will be huge and the efficiency will be low. Channel is a logical connection established inside the connection. If the application supports multi-threading, usually each thread creates a separate channel for communication. AMQP method includes the channel id to help the client and message broker identify the channel, so the channels are completely isolated of. As a lightweight Connection, Channel greatly reduces the overhead of the operating system to establish a TCP connection

• Exchange

  When the message arrives at the first stop of the broker, it matches the routing key in the query table according to the distribution rules, and distributes the message to the queue. Commonly used types are: direct (point-to-point), topic (publish-subscribe) and fanout (multicast)

• Queue

  The message is finally sent here to be picked up by the consumer

• Binding

  The virtual connection between the exchange and the queue, the binding can contain the routing key, and the Binding information is saved in the query table in the exchange, which is used for the distribution basis of the message

RabbitMQ installation

This time, we will only explain the installation method of rpm and the configuration of roles under the Linux system:

Since the bottom layer of RabbitMQ is written in erlang language, it is necessary to install the environment of erlang language when installing RabbitMQ. The following is the address of the official website:

RabbitMQ official website download address: https://www.rabbitmq.com/download.html

Erlang official website download address: https://www.erlang-solutions.com/downloads/

After uploading to the linux system, install it:

#  erlangxxxxx.rpm是erlang完整的名称
rpm -ivh erlangxxxxx.rpm
yum install socat -y
# rabbitmqxxxxx.rpm是rabbitMQ完整的名称
rpm -ivh rabbitmqxxxxx.rpm

Common commands for RabbitMQ:

# 启动服务
systemctl start rabbitmq-server
# 查看服务状态
systemctl status rabbitmq-server 
# 开启自启动
systemctl enable rabbitmq-server
# 停止服务
systemctl stop rabbitmq-server 
# 重启服务
systemctl restart rabbitmq-server

Web management interface

RabbitMQ has its own web management interface, and the default port used is 15672

The first web management interface using RabbitMQ needs to install the web client plug-in:

rabbitmq-plugins enable rabbitmq_management

After the installation is complete, restart the RabbitMQ service

systemctl restart rabbitmq-server

Enter the web management interface of RabbitMQ, we can see that we need to log in to the account.

RabbitMQ comes with a guest account, but it needs to be configured to log in to the guest account. So we need to recreate our own account, the command is as follows:

# 创建账号
rabbitmqctl add_user admin 123
# 设置用户的角色
rabbitmqctl set_user_tags  admin administrator 
# 设置用户的权限
# set_permissions [-p <vhostpath>] <user> <conf> <write> <read>
rabbitmqctl set_permissions -p "/" admin ".*" ".*" ".*"

User roles mainly include the following four types:

1. administrator : You can log in to the console, view all information, and manage rabbitmq
2. monitoring : The monitor logs in to the console to view all information
3. policymaker : The policy maker logs in to the console and specifies the policy
4. managment : ordinary administrators log in to the console

Open it again, log in with the account admin and password 123, and enter the background interface of RabbitMQ after successful login:

Getting started

Hello World

In this part of the tutorial, we'll write two programs in Java. A producer that sends a single message and receives a message and prints

consumers out. We'll cover some of the details in the Java API.

In the diagram below, "P" is our producer and "C" is our consumer. The middle box is a queue - RabbitMQ generation

The message buffer kept by the table consumer.

After creating the project in idea, import the required dependencies.

<!--指定 jdk 编译版本-->
<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <configuration>
                <source>8</source>
                <target>8</target>
            </configuration>
        </plugin>
    </plugins>
</build>
<dependencies>
    <!--rabbitmq 依赖客户端-->
    <dependency>
        <groupId>com.rabbitmq</groupId>
        <artifactId>amqp-client</artifactId>
        <version>5.8.0</version>
    </dependency>
    <!--操作文件流的一个依赖-->
    <dependency>
        <groupId>commons-io</groupId>
        <artifactId>commons-io</artifactId>
        <version>2.6</version>
    </dependency>
</dependencies>

message producer code

package com.it.rabbitmq;

import com.rabbitmq.client.Channel;
import com.rabbitmq.client.ConnectionFactory;

import java.nio.charset.StandardCharsets;

public class Producer {
    
    
    public static void main(String[] args) throws Exception{
    
    
        //创建一个连接工厂
        ConnectionFactory connectionFactory = new ConnectionFactory();
        //设置要连接RabbitMQ所在服务器的ip地址，这里要进行填写自己的
        connectionFactory.setHost("121.56.241.29");
        //设置账号
        connectionFactory.setUsername("admin");
        //设置密码
        connectionFactory.setPassword("123");
        //创建信道channel
        //channel实现了自动close接口，自动关闭
        Channel channel = connectionFactory.newConnection().createChannel();
        /**
         * 进行生成一个队列
         * 1. 队列名称
         * 2. 队列里面是否持久化
         * 3. 该队列是否只进行提供一个消费者进行（是否共享）
         * 4. 是否自动删除，即最后一个消费者断开连接之后，队列是否自动删除该消息
         * 5. 其他参数
         */
        //队列名称为test，不持久化，不共享，不自动删除，没有其他参数
        channel.queueDeclare("test",false,false,false,null);
        String msg = "Hello World";
        /**
         * 进行发送一个消息
         * 1. 发送到哪个交换机（目前没有交换机进行传空，代表的是默认交换机）
         * 2. 路由的key（当前的key没有声明，使用队列名称即可）
         * 3. 其他参数信息
         * 4. 发送消息的消息体
         */
        channel.basicPublish("","test",null,msg.getBytes(StandardCharsets.UTF_8));
        System.out.println("消息发送成功");
    }
}

message consumer code

package com.it.rabbitmq;

import com.rabbitmq.client.CancelCallback;
import com.rabbitmq.client.Channel;
import com.rabbitmq.client.ConnectionFactory;
import com.rabbitmq.client.DeliverCallback;

public class Consumer {
    
    
    public static void main(String[] args) throws Exception{
    
    
        //创建连接工厂
        ConnectionFactory connectionFactory = new ConnectionFactory();
        //设置ip地址
        connectionFactory.setHost("121.56.241.29");
        //设置账号
        connectionFactory.setUsername("admin");
        //设置密码
        connectionFactory.setPassword("123");
        //获取信道
        Channel channel = connectionFactory.newConnection().createChannel();
        //消息成功接受的回调方法,采用lambda表达式
        DeliverCallback deliverCallback = (tag,message)->{
    
    
            //将信道存储的信息转化为字符串类型
            String msg = new String(message.getBody());
            System.out.println(message);
        };
        //取消消费的回调接口
        CancelCallback cancelCallback = (tag)->{
    
    
            System.out.println("消息被取消");  
        };
        /**
         * 消费者消费消息
         * 1. 消费哪个队列
         * 2. 消费成功后是否自动应答
         * 3. 消费成功的回调接口
         * 4. 消费取消的回调接口
         */
        channel.basicConsume("test",true,deliverCallback,cancelCallback);
    }
}

First, the producer must be run, create a queue in RabbieMQ, and then run the producer. The effect is as follows:

Work Queue

Work Queues - The main idea of ​​Work Queues (also known as Task Queues) is to avoid executing resource-intensive tasks immediately and having to wait for it to complete. Instead we schedule the task to be executed later. We encapsulate the task as a message and send it to the queue. A worker process running in the background will pop the tasks and eventually execute the job. When there are multiple worker threads, these worker threads will work on these tasks together.

Polling to send messages

In this case, we will start two worker threads, one message sending thread, let's see how their two worker threads work.

extract work class

package com.it.rabbitmq.util;

import com.rabbitmq.client.Channel;
import com.rabbitmq.client.ConnectionFactory;

public class RabbitMQUtil {
    
    
    //进行获取信道channel
    public static Channel getChannel() throws Exception{
    
    
        //创建连接工厂
        ConnectionFactory connectionFactory = new ConnectionFactory();
        //设置ip地址
        connectionFactory.setHost("121.56.241.29");
        //设置账号
        connectionFactory.setUsername("admin");
        //设置密码
        connectionFactory.setPassword("123");
        //获取信道
        Channel channel = connectionFactory.newConnection().createChannel();
        return channel;
    }
}

Start a sending thread

package com.it.rabbitmq;

import com.it.rabbitmq.util.RabbitMQUtil;
import com.rabbitmq.client.Channel;

import java.nio.charset.StandardCharsets;

//发送线程，生产者
public class Producer {
    
    
    public static void main(String[] args) throws Exception{
    
    
        Channel channel = RabbitMQUtil.getChannel();
        /**
         * 进行生成一个队列
         * 1. 队列名称
         * 2. 队列里面是否持久化
         * 3. 该队列是否只进行提供一个消费者进行（是否共享）
         * 4. 是否自动删除，即最后一个消费者断开连接之后，队列是否自动删除该消息
         * 5. 其他参数
         */
        //队列名称为test，不持久化，不共享，不自动删除，没有其他参数
        channel.queueDeclare("test",false,false,false,null);
        String msg = "Hello World";
        /**
         * 进行发送一个消息
         * 1. 发送到哪个交换机（目前没有交换机进行传空，代表的是默认交换机）
         * 2. 路由的key（当前的key没有声明，使用队列名称即可）
         * 3. 其他参数信息
         * 4. 发送消息的消息体
         */
        channel.basicPublish("","test",null,msg.getBytes(StandardCharsets.UTF_8));
        System.out.println("消息发送成功");
    }
}

Start a consumer thread

package com.it.rabbitmq;

import com.it.rabbitmq.util.RabbitMQUtil;
import com.rabbitmq.client.CancelCallback;
import com.rabbitmq.client.Channel;
import com.rabbitmq.client.DeliverCallback;

//接受线程消费者
public class Consumer {
    
    
    public static void main(String[] args) throws Exception{
    
    
        Channel channel = RabbitMQUtil.getChannel();
        //消息成功接受的回调方法,采用lambda表达式
        DeliverCallback deliverCallback = (tag,message)->{
    
    
            //将信道存储的信息转化为字符串类型
            String msg = new String(message.getBody());
            System.out.println("接收到的消息："+msg);
        };
        //取消消费的回调接口
        CancelCallback cancelCallback = (tag)->{
    
    
            System.out.println("消息被取消");
        };
        /**
         * 消费者消费消息
         * 1. 消费哪个队列
         * 2. 消费成功后是否自动应答
         * 3. 消费成功的回调接口
         * 4. 消费取消的回调接口
         */
        channel.basicConsume("test",true,deliverCallback,cancelCallback);
    }
}

The result of the operation is as follows:

message reply

Consumers can take a while to complete a task, what happens if one of the consumers processes a long task and only completes part of it and suddenly it dies. RabbitMQ marks a message for deletion as soon as it delivers it to a consumer. In this case, suddenly a consumer dies and we lose the messages we were processing. and subsequent messages sent to the consumer because it was unable to receive them.

In order to ensure that the message is not lost during the sending process, a message response mechanism is introduced. The message response is: after the consumer receives the message and processes the message, it tells rabbitmq that it has been processed, and rabbitmq can delete the message.

auto answer

Immediately after the message is sent, it is considered to have been successfully transmitted. This mode requires a trade-off between high throughput and data transmission security , because in this mode, if the message is received before the message is received, there is a connection on the consumer side or the channel is closed, then the message It is lost. Of course, on the other hand, consumers in this mode can transmit overloaded messages without limiting the number of messages transmitted . Of course, this may cause consumers to receive too many messages that are too late to process, resulting in The backlog of these messages will eventually cause memory to run out, and eventually these consumer threads will be killed by the operating system, so this pattern is only suitable if the consumer can process these messages efficiently and at a certain rate.

Manual answer method

• Channel.basicAck (for positive confirmation)

  RabbitMQ already knows about the message and successfully processed it, so it can be discarded

• Channel.basicNack (for negative acknowledgment)

• Channel.basicReject (for negative confirmation)

  Compared with Channel.basicNack, there is one less parameter. If the message is not processed, it will be directly rejected, and it can be discarded.

The advantage of manual answering is that it can be answered in batches and reduce network congestion

Subsequent content will continue to be updated, click free follow to not get lost.