Introduction and configuration of message queue

1. Detailed explanation of message queue

Message Queue ( Message Queue ) is an important component in a distributed system . Its general usage scenario can be simply described as: when the results do not need to be obtained immediately, but the concurrency needs to be controlled, the Message Queue is almost needed. when

Mainly solves: application coupling , asynchronous messaging , traffic shaving and other issues. Achieve high performance, high availability, scalability and eventually consistent architecture. It is an indispensable middleware for large-scale distributed systems.

Currently in production environments, the most commonly used message queues include ActiveMQ , RabbitMQ, ZeroMQ, Kafka, MetaMQ, RocketMQ, etc.

1. Two models of message queues

1.1. Point-to-point model

Each message has only one receiver ( Consumer ). Once it is messaged, it is no longer in the message queue.
There is no dependency between the sender and the receiver. After the sender sends the message, no matter whether there is a receiver running or not, it will not affect the following. Send once

img

1.2. Publish and subscribe model

Each message can have multiple subscribers.
Each subscriber can receive all messages of the topic.

img

The difference between the two models: whether a message can be consumed multiple times.
If there is only one subscriber, the two models are basically the same, so the publish-subscribe model is compatible with the queue model at the functional level.

2. Message queue implements distributed transactions

A strict transaction implementation has four ACID properties: atomicity, consistency, isolation, and durability.
Atomicity: a transaction operation is indivisible. Either all succeed or all fail. Half of them succeed and half of them fail.
Consistency : At the point in time before the transaction execution is completed, the data read must be the data before the update, and the data read after the update must be the data after the update. Isolation
: the execution of a transaction cannot be interfered by other transactions (the operations within a transaction and the data used The data is isolated from other ongoing transactions, and concurrently executed transactions cannot interfere with each other.)
Durability: Once a transaction is submitted, subsequent operations and failures will not have any impact on the outcome of the transaction.

In a distributed system, it is very difficult to achieve data consistency alone, so it is generally only guaranteed to achieve ultimate consistency. The more common distributed transaction implementations include

1), 2PC ( Two-phase Commit ) two-phase commit
2), TCC ( Try-Confirm-Cancel )
3), transaction message

3. Message queue distributed transaction implementation

Both Kafka and RocketMQ provide transaction-related functions. Let’s take orders as an example to see how to implement them.

img

Both Kafka and RocketMQ provide transaction-related functions. The core: half message and
half message: This half message does not mean that the message content is incomplete, but means that before the transaction is submitted, the message is invisible to the consumer ( sendMessageInTransaction )

(1) Application scenarios (or functions) of message queues

Application scenarios are divided into: asynchronous processing , application decoupling , traffic cutting , message communication , etc.; the most important ones are: asynchronous processing , application decoupling , traffic cutting

1. Asynchronous processing

Scenario: After the user registers, it is necessary to send registration emails and registration information. There are two traditional methods: serial method and parallel method.

Disadvantages of the traditional model: some non-essential business logic runs in a synchronous manner, which is too time-consuming.
Advantages of the middleware model: writes messages to the message queue, and non-essential business logic runs asynchronously, speeding up the response.

1.1. Serial mode

After the registration information is successfully written into the database, a registration email is sent, and then a registration SMS is sent. After all tasks are completed, the information is returned to the client; as shown in the figure

img

1.2. Parallel method

After the registration information is successfully written to the database, the registration email and registration SMS are sent at the same time. After all tasks are executed, information is returned to the client. Compared with serial mode, parallel mode can improve execution efficiency and reduce execution time.

img

Assume that all three operations require 50ms of execution time. Excluding network factors, the final execution is completed. The serial mode requires 150ms, while the parallel mode requires 100ms
because the number of requests processed by the CPU in unit time is the same. Assume: CPU every 1 The second throughput is 100 times, then the number of requests that can be executed in 1 second in serial mode is 1000/150, which is less than 7 times; the number of requests that can be executed in 1 second in parallel mode is 1000/100, which is 10 times. It can be seen
that , the traditional serial and parallel methods will be limited by system performance, so how to solve this problem?
We need to introduce a message queue to process non-essential business logic asynchronously. The resulting process is as follows:

img

According to the above process, the user's response time is basically equivalent to the time it takes to write user data to the database. After sending registration emails and sending registration SMS messages, the execution results can be returned after being written to the message queue. The time it takes to write the message queue is very short. It is fast and can be almost ignored. This can also increase the system throughput to 20QPS ( query rate per second ), which is nearly 3 times higher than the serial mode and 2 times higher than the parallel mode.

QPS: Query rate per second, which is a measure of how much traffic a specific query server handles within a specified period of time.

On the Internet, the query rate per second is often used to measure the performance of the domain name system server machine, which is the QPS corresponding to fetches/sec , which is the number of response requests per second, which is the maximum throughput capacity.

2. Application decoupling

Scenario: A certain system A needs to deal with other systems (that is, call methods). If other systems change or new systems are added, system A will also change. In this case, the coupling degree is relatively high and it is more troublesome.

img

Middleware pattern: Use message queues to solve this problem

Our A system sends the generated data to the message queue, and other systems go to the message queue for consumption. Then the reduction or addition of other systems has little to do with the A system, so as to achieve the decoupling function.

img

3. Traffic sharpening

For example, in the scenario of product flash sale business, the traffic will usually increase sharply and the application will crash due to excessive traffic. To solve this problem, it is generally necessary to add a message queue to the application front-end.

1), can control the number of people participating in the event
2), can alleviate the huge pressure on applications caused by high traffic in a short period of time

The processing method is as shown in the figure:

img

1) After receiving the user request, the server first writes to the message queue. At this time, if the number of messages in the message queue exceeds the maximum number, the user request will be directly rejected or the error page will be returned.
2) The flash sale business reads the request information in the message queue according to the flash sale rules and performs subsequent processing.

4. Message communication

Log processing refers to using message queues in log processing, such as Kafka applications, to solve the problem of large number of log transmissions

img

Log collection client: Responsible for log data collection, scheduled writing and writing to Kafka queue
Kafka message queue: Responsible for receiving, storing and forwarding log data
Log processing application: Subscribing and consuming log data in the Kafka queue

(2) Disadvantages of using message queues

1. System availability is reduced: System availability is reduced to some extent. Why do you say this? Before joining MQ, you don't have to think about message loss or MQ hanging up, etc. However, after introducing MQ, you need to think about it!
2. Increased system complexity: After joining MQ, you need to ensure that messages are not consumed repeatedly, deal with message loss, ensure the order of message delivery, and other issues!
3. Consistency issue: Message queue can achieve asynchronous, and the asynchronous brought by message queue can indeed improve the system response speed. But what if the real consumer of the message didn’t consume the message correctly? This will lead to data inconsistency.

(3) Common message queues

Comparison of common message queues
ActiveMQ RabbitMQ RocketMQ kafka
Development language java erlang java scala
Single machine throughput Level 10,000 Level 10,000 One hundred thousand level One hundred thousand level
Timeliness ms level us level ms level Within ms level
Availability High (master-slave architecture) High (master-slave architecture) Very high (distributed architecture) Very high (distributed architecture)
Features Mature product, used in many companies; has more documentation; has good support for various protocols Developed based on Erlang, it has strong concurrency capabilities, extremely good performance, low latency, and rich management interfaces. MQ has relatively complete functions and good scalability Only the main MQ functions are supported. Some message query, message traceback and other functions are not provided. After all, it is prepared for big data and is widely used in the field of big data.

Based on the above comparison, the following conclusions can be drawn

1) For small and medium-sized software companies, it is recommended to choose RabbitMQ
because the erlang language is inherently characterized by high concurrency, and its management interface is very convenient to use. As the saying goes, Xiao He is also a success, and Xiao He is a failure! Its shortcomings are also here. Although RabbitMQ is open source, how many programmers in China can customize and develop Erlang ? Fortunately, the RabbitMQ community is very active and can solve bugs encountered during the development process . This is very important for small and medium-sized companies. The reason why kafka
is not considered is that small and medium-sized software companies are not as good as Internet companies and the amount of data is not that large. Message middleware should be preferred with relatively complete functions, so Kafka excludes rocketmq . The reason for not considering rocketmq is that rocketmq is produced by Alibaba. If Alibaba gives up maintaining rocketmq , small and medium-sized companies generally cannot spare people to carry out customized development of rocketmq , so they will not recommend

2) Large software companies should choose between rocketMQ and kafka based on specific applications
. Large software companies have enough funds to build a distributed environment and a large enough amount of data. For rocketMQ , large software companies can also dedicate manpower to customized development of rocketMQ . After all , there are still quite a few people in China who have the ability to modify JAVA source code. As for kafka , depending on the business scenario, if it has a log collection function, kafka is definitely the first choice . Which one to choose depends on the usage scenario.

(4) High availability of message queue

The rcoketMQ cluster has multi- master mode, multi- master and multi- slave asynchronous replication mode, and multi- master and multi- slave synchronous dual-write mode, as shown in the figure:

img

Producer establishes a long connection with one of the nodes (randomly selected) in the NameServer cluster, regularly obtains Topic routing information from NameServer , establishes a long connection with the Broker Master that provides Topic services , and regularly sends heartbeats to the Broker . The Producer can only send messages to the Broker master , but the Consumer is different. It establishes long-term connections with the Master and Slave that provide Topic services at the same time. It can subscribe to messages from the Broker Master or from the Broker Slave .

kafka

img

A typical Kafka cluster contains several Producers (which can be Page View generated by the web front-end , or server logs, system CPU , Memory , etc.), several brokers ( Kafka supports horizontal expansion. Generally, the greater the number of brokers , the higher the cluster throughput rate. ), several Consumer Groups , and a Zookeeper cluster. Kafka uses Zookeeper to manage cluster configuration, elect leaders , and rebalance when the Consumer Group changes . Producer uses push mode to publish messages to broker , and Consumer uses pull mode to subscribe and consume messages from broker .

rabbitMQ: There are also ordinary cluster and mirror cluster modes, which I won’t explain in detail here.

(5) Repeated consumption of messages

The reason for repeated consumption: When the consumer consumes a message, after the consumption is completed, it will send a confirmation message to the message queue. The message queue will know that the message has been consumed and will delete the message from the message queue. It’s just that different message queues send different forms of confirmation information. For example, RabbitMQ sends an ACK confirmation message, and RocketMQ returns a CONSUME_SUCCESS success flag. Kafka actually has the concept of offset . To put it simply, every message has an offset . After kafka consumes the message, it needs to submit the offset to let the message queue know that it has consumed it. What is the reason for repeated consumption? It is because of network transmission and other failures, the confirmation information is not transmitted to the message queue, causing the message queue to not know that it has consumed the message and distribute the message to other consumers again.

Solution:
1. For example: you get this message and perform an insert operation in the database. That would be easy. Give this message a unique primary key. Then even if there is repeated consumption, it will cause a primary key conflict and avoid dirty data in the database. 2. For example: if you get this
message and perform a redis set operation, then It's easy and there is no need to solve it, because the result will be the same no matter how many times you set it. The set operation is considered an idempotent operation.
3. Prepare a third-party medium: for consumption records. Taking redis as an example, assign a global id to the message. As long as the message is consumed, <id, message> will be written to redis in the form of KV. Before the consumer starts consuming, he can first check whether there is any consumption record in redis.

(6) Sexual transmission of consumption

Each type of MQ must be analyzed from three perspectives: the producer loses data, the message queue loses data, and the consumer loses data.

RabbitMQ

1. Producer loses data

From the perspective of the producer losing data, RabbitMQ provides transaction and confirm modes to ensure that the producer does not lose messages.
The transaction mechanism means that before sending the message, open the transaction ( channel.txSelect() ), and then send the message. If the sending process If any exception occurs in the transaction, the transaction will be rolled back ( channel.txRollback() ). If the transmission is successful, the transaction will be submitted ( channel.txCommit() ).
However, the disadvantage is that the throughput is reduced. Therefore, confirm mode is mostly used in production . Once the channel enters confirm mode, all messages published on the channel will be assigned a unique ID (starting from 1). Once the message is delivered to all matching queues, rabbitMQ will send an Ack to the producer ( Contains the unique ID of the message ), which allows the producer to know that the message has correctly arrived at the destination queue. If rabiitMQ fails to process the message, it will send a Nack message to you, and you can retry the operation.

2. The message queue loses data

To deal with the situation of message queue losing data, it is usually to enable the configuration of persistent disk. This persistence configuration can be used in conjunction with the confirm mechanism. You can send an Ack signal to the producer after the message is persisted to disk. In this way, if rabbitMQ dies before the message is persisted to the disk, then the producer cannot receive the Ack signal and the producer will automatically resend it
. So how to persist it? It is actually very easy. Just follow the following two steps:
1. Persistence of the queue The flag durable is set to **true, ** represents a durable queue
. 2. Set deliveryMode =2 when sending a message
. After setting deliveryMode =2 like this, even if rabbitMQ hangs, the data can be restored after restarting.

3. Consumers lose data

Consumers lose data generally because they use the automatic confirmation message mode. In this mode, consumers automatically acknowledge receipt of the message. At this time, rahbitMQ will immediately delete the message. In this case, if the consumer encounters an exception and fails to process the message, the message will be lost.
As for the solution, just confirm the message manually.

2. RabbitMQ installation

1. Install Erlang

Official download of Erlang: Downloads - Erlang/OTP

img

 
[root@servers ~]# yum -y install make gcc gcc-c++ kernel-devel m4 ncurses-devel openssl-devel                     
[root@servers ~]# tar xvf otp_src_24.2-rc1.tar.gz                     
[root@servers ~]# cd otp_src_24.2                 
[root@servers otp_src_24.2]# ./configure --prefix=/usr/local/erlang --with-ssl -enable-threads -enable-smmp-support -enable-kernel-poll --enable-hipe --without-javac                         
[root@servers otp_src_24.2]# make && make install
[root@servers otp_src_24.2]# vim /etc/profile                 
................      在最后加入
........
ERLANG_HOME=/usr/local/erlang
PATH=$ERLANG_HOME/bin:$PATH
export ERLANG_HOME
export PATH
 
保存
 
root@servers ~]# source /etc/profile            
[root@servers ~]# erl              #验证是否安装成功
Erlang/OTP 24 [erts-12.0] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:1]
 
Eshell V12.0  (abort with ^G)
1> 
 
 
ctrl + C  退出  如果一次没有退出就多按几次

2. Install RadditMQ

Download RabbitMQ: https://github.com/rabbitmq/rabbitmq-server/releases/tag/v3.9.12

img

 
[root@servers ~]# xz -d rabbitmq-server-generic-unix-3.9.12.tar.xz                          
[root@servers ~]# tar xf rabbitmq-server-generic-unix-3.9.12.tar                   
[root@servers ~]# cp -rf rabbitmq_server-3.9.12 /usr/local/               
[root@servers ~]# cd /usr/local/                  
[root@servers local]# mv rabbitmq_server-3.9.12 rabbitmq                        
[root@servers local]# cd rabbitmq/sbin/                   
 
[root@servers sbin]# ./rabbitmq-plugins enable rabbitmq_management        #开启管理页面插件
Enabling plugins on node rabbit@C7--13:
rabbitmq_management
The following plugins have been configured:
  rabbitmq_management
  rabbitmq_management_agent
  rabbitmq_web_dispatch
Applying plugin configuration to rabbit@C7--13...
The following plugins have been enabled:
  rabbitmq_management
  rabbitmq_management_agent
  rabbitmq_web_dispatch
 
set 3 plugins.
Offline change; changes will take effect at broker restart.

Start and shut down services

 
[root@servers sbin]# ./rabbitmq-server                  #启动,ctrl + c 退出及关闭
 
[root@servers sbin]# ./rabbitmq-server -detached        #在后台启动服务
 
[root@servers sbin]# ./rabbitmqctl stop                 #关闭服务

Add administrator account

Add users after starting in the background

 
     添加用户格式:    ./rabbitmqctl add_user 用户名 密码
 
[root@servers sbin]# ./rabbitmqctl add_user admin 123.com               
Adding user "admin" ...
Done. Don't forget to grant the user permissions to some virtual hosts! See 'rabbitmqctl help set_permissions' to learn more.
    分配用户标签格式: ./rabbitmqctl add_user_tags 用户名 管理员标签[administrator]
[root@servers sbin]# ./rabbitmqctl set_user_tags admin administrator            
Setting tags for user "admin" to [administrator] ...

access test

img

Enter the previously set user: admin and password: 123.com to log in

img

3. Detailed explanation of RabbitMQ

(1) Introduction to AMQP

RabbitMQ is an open source message broker software (also known as message-oriented middleware) that implements the Advanced Message Queuing Protocol ( AMQP protocol ). The RabbitMQ server is written in the Erlang language, while clustering and failover are built on the Open Telecommunications Platform framework. All major programming languages ​​have client libraries that communicate with the proxy interface

AMQPProtocol : Advanced Message Queuing Protocol is an open standard for application layer protocols and is designed for message-oriented middleware. Message middleware is mainly used for decoupling between components. The sender of a message does not need to know the existence of the message consumer, and vice versa. The
AMQP main features of the protocol are message-oriented, queues, routing (including point-to-point and publish/subscribe), and reliability. , Secure
RabbitMQ is an open source
AMQP
implementation. The server is written in Erlang language and supports a variety of clients, such as: Python , Ruby , .NET , Java , JMS , C , PHP , ActionScript , XMPP , STOMP , etc., and supports AJAX . It is used to store and forward messages in distributed systems, and performs well in terms of ease of use, scalability, and high availability.

AMQP three-layer protocol
Module Layer Located at the highest level of the protocol, it mainly defines some commands for the client to call. The client can use these commands to implement its own business logic. For example, the client can declare a queue through queue.declare and use the consume command to obtain messages in a queue .
Session Layer It is mainly responsible for sending the client's commands to the server, and returning the server's response to the client. It is mainly responsible for providing reliability, synchronization mechanism and error handling for the communication between the client and the server.
Transport Layer It mainly transmits binary data streams and provides frame processing, channel multiplexing, error detection and data representation.

(2) Function

store and forward Multiple message senders, single message receiver
Distributed transactions Multiple message senders, multiple message receivers
publishsubscribe Multiple message senders, multiple message receivers
content-based routing Multiple message senders, multiple message receivers
file transfer queue Multiple message senders, multiple message receivers
point-to-point connection Single message sender, single message receiver

(3) Terminology description

AMQP model ( AMQP Model) A logical framework represented by the key entities and semantics that an AMQP- compliant server must provide. To implement the semantics defined in this specification, clients can send commands to control the AMQP server
connect( Connection) A network connection, such as a TCP/IP socket connection
session( Session) Named conversations between endpoints. Within a session context, guaranteed to be delivered "exactly once"
channel( Channel) An independent bidirectional data flow channel within a multiplexed connection. Provide a physical transmission medium for the session
client( Client) The initiator of the AMQP connection or session. AMQP is asymmetric, clients produce and consume messages, and servers store and route these messages.
server( Server) A process that accepts client connections and implements AMQP message queue and routing functions. Also called a "message broker"
endpoint( Peer) Either party to the AMQP conversation. An AMQP connection consists of two endpoints (one is the client and the other is the server)
Partner ( Partner) When describing the interaction between two endpoints, the term "partner" is used as a shorthand for the "other" endpoint. For example, we define endpoint A and endpoint B. When they communicate, endpoint B is the partner of endpoint A, and endpoint A is the partner of endpoint B.
fragment set ( Assembly) An ordered collection of segments forming a logical unit of work
segment( Segment) An ordered collection of frames, forming a complete subunit of the fragment set
frame( Frame) An atomic unit of AMQP transport. A frame is any fragment in a segment
control( Control) One-way instructions, the AMQP specification assumes that the transmission of these instructions is unreliable
command( Command) Instructions that require confirmation. The AMQP specification stipulates that the transmission of these instructions is reliable.
Exception( Exception) Error status that may occur when executing one or more commands
class ( Class) A batch of AMQP commands or controls used to describe a specific function
Message header ( Header) A special section that describes message data attributes
Message body ( Body) 包含应用程序数据的一种特殊段。消息体段对于服务器来说完全透明——服务器不能查看或者修改消息体
消息内容(Content 包含在消息体段中的的消息数据
交换器(Exchange 服务器中的实体,用来接收生产者发送的消息并将这些消息路由给服务器中的队列
交换器类型(Exchange Type 基于不同路由语义的交换器类
消息队列(Message Queue 一个命名实体,用来保存消息直到发送给消费者
绑定器(Binding 消息队列和交换器之间的关联
绑定器关键字(Binding Key 绑定的名称。一些交换器类型可能使用这个名称作为定义绑定器路由行为的模式
路由关键字(Routing Key 一个消息头,交换器可以用这个消息头决定如何路由某条消息
持久存储(Durable 一种服务器资源,当服务器重启时,保存的消息数据不会丢失
临时存储(Transient 一种服务器资源,当服务器重启时,保存的消息数据会丢失
持久化(Persistent 服务器将消息保存在可靠磁盘存储中,当服务器重启时,消息不会丢失
非持久化(Non-Persistent 服务器将消息保存在内存中,当服务器重启时,消息可能丢失
消费者(Consumer 一个从消息队列中请求消息的客户端应用程序
生产者(Producer 一个向交换器发布消息的客户端应用程序
虚拟主机(Virtual Host 一批交换器、消息队列和相关对象。虚拟主机是共享相同的身份认证和加密环境的独立服务器域。客户端应用程序在登录到服务器之后,可以选择一个虚拟主机

(四)、RabbitMQ的工作流程

消息队列有三个概念: 发消息者、消息队列、收消息者。RabbitMQ 在这个基本概念之上, 多做了一层抽象, 在发消息者和队列之间, 加入了交换器 (Exchange)。这样发消息者和消息队列就没有直接联系,转而变成发消息者把消息发给交换器,交换器根据调度策略再把消息转发给消息队列
消息生产者并没有直接将消息发送给消息队列,而是通过建立与ExchangeChannel,将消息发送给ExchangeExchange根据路由规则,将消息转发给指定的消息队列。消息队列储存消息,等待消费者取出消息。消费者通过建立与消息队列相连的Channel,从消息队列中获取消息

img

Producer(消息的生产者) 向消息队列发布消息的客户端应用程序
Channel(信道) 多路复用连接中的一条独立的双向数据流通道。信道是建立在真实的TCP连接内的虚拟连接,复用TCP连接的通道
Routing Key(路由键) 消息头的一个属性,用于标记消息的路由规则,决定了交换机的转发路径。最大长度255 字节
Broker RabbitMQ Server,服务器实体
Binding(绑定) 用于建立ExchangeQueue之间的关联。一个绑定就是基于Binding KeyExchangeQueue连接起来的路由规则,所以可以将交换器理解成一个由Binding构成的路由表
Exchange(交换器|路由器) 提供ProducerQueue之间的匹配,接收生产者发送的消息并将这些消息按照路由规则转发到消息队列。交换器用于转发消息,它不会存储消息 ,如果没有 Queue绑定到 Exchange 的话,它会直接丢弃掉 Producer 发送过来的消息。交换器有四种消息调度策略,分别是fanout, direct, topic, headers
Binding Key(绑定键) ExchangeQueue的绑定关系,用于匹配Routing Key。最大长度255 字节
Queue(消息队列) 存储消息的一种数据结构,用来保存消息,直到消息发送给消费者。它是消息的容器,也是消息的终点。一个消息可投入一个或多个队列。消息一直在队列里面,等待消费者连接到这个队列将消息取走。需要注意,当多个消费者订阅同一个Queue,这时Queue中的消息会被平均分摊给多个消费者进行处理,而不是每个消费者都收到所有的消息并处理,每一条消息只能被一个订阅者接收
Consumer(消息的消费者) 从消息队列取得消息的客户端应用程序
Message(消息) 消息由消息头和消息体组成。消息体是不透明的,而消息头则由一系列的可选属性组成,这些属性包括routing-key(路由键)、priority(消息优先权)、delivery-mode(是否持久性存储)等

(五)、Exchange消息调度策略

交换器的功能主要是接收消息并且转发到绑定的队列,交换器不存储消息,在启用ack模式后,交换器找不到队列会返回错误

调度策略是指Exchange在收到生产者发送的消息后依据什么规则把消息转发到一个或多个队列中保存。调度策略与三个因素相关:Exchange TypeExchange的类型),Binding KeyExchangeQueue的绑定关系),消息的标记信息(Routing Keyheaders
Exchange根据消息的Routing Key和Exchange绑定QueueBinding Key分配消息。生产者在将消息发送给Exchange的时候,一般会指定一个Routing Key,来指定这个消息的路由规则,而这个Routing Key需要与Exchange TypeBinding Key联合使用才能最终生效
Exchange TypeBinding Key固定的情况下(一般这些内容都是固定配置好的),我们的生产者就可以在发送消息给Exchange时,通过指定Routing Key来决定消息流向哪里

交换器的四种消息调度策略:fanout, direct, topic, headers

1、Fanout (订阅模式**/**广播模式)

img

交换器会把所有发送到该交换器的消息路由到所有与该交换器绑定的消息队列中。订阅模式
Binding KeyRouting Key无关,交换器将接受到的消息分发给有绑定关系的所有消息队列队列(不论Binding KeyRouting Key是什么)。类似于子网广播,子网内的每台主机都获得了一份复制的消息。Fanout交换机转发消息是最快的

img

2、Direct(路由模式)

img

精确匹配:当消息的Routing KeyExchangeQueue 之间的Binding Key完全匹配,如果匹配成功,将消息分发到该Queue。只有当Routing KeyBinding Key完全匹配的时候,消息队列才可以获取消息。DirectExchange的默认模式
RabbitMQ默认提供了一个Exchange,名字是空字符串,类型是Direct,绑定到所有的Queue(每一个Queue和这个无名Exchange之间的Binding KeyQueue的名字)。所以,有时候我们感觉不需要交换器也可以发送和接收消息,但是实际上是使用了RabbitMQ默认提供的Exchange

img

3、Topic (通配符模式)

img

按照正则表达式模糊匹配:用消息的Routing KeyExchangeQueue 之间的Binding Key进行模糊匹配,如果匹配成功,将消息分发到该Queue
Routing Key是一个句点号“. ”分隔的字符串(我们将被句点号“. ”分隔开的每一段独立的字符串称为一个单词)。Binding KeyRouting Key一样也是句点号“. ”分隔的字符串。Binding Key中可以存在两种特殊字符“ * ”与“#”,用于做模糊匹配,其中“*”用于匹配一个单词,“#”用于匹配多个单词(可以是零个)

img

4、Headers(键值对模式)

Headers不依赖于Routing KeyBinding Key的匹配规则来转发消息,交换器的路由规则是通过消息头的Headers属性来进行匹配转发的,类似HTTP请求的Headers
在绑定QueueExchange时指定一组键值对,键值对的Hash结构中要求携带一个键“x-match”,这个键的Value可以是anyall,代表消息携带的Hash是需要全部匹配(all),还是仅匹配一个键(any)
当消息发送到Exchange时,交换器会取到该消息的headers,对比其中的键值对是否完全匹配QueueExchange绑定时指定的键值对;如果完全匹配则消息会路由到该Queue,否则不会路由到该QueueHeaders交换机的优势是匹配的规则不被限定为字符串(String),而是Object类型

img

(六)、RPC(Remote Procedure Call,远程过程调用)

img

MQ本身是基于异步的消息处理,前面的示例中所有的生产者(P)将消息发送到RabbitMQ后不会知道消费者(C)处理成功或者失败(甚至连有没有消费者来处理这条消息都不知道)
但实际的应用场景中,我们很可能需要一些同步处理,需要同步等待服务端将我的消息处理完成后再进行下一步处理。这相当于RPCRemote Procedure Call,远程过程调用)。在RabbitMQ中也支持RPC

img

RabbitMQ中实现RPC的机制是:
1、客户端发送请求(消息)时,在消息的属性(MessageProperties,在AMQP协议中定义了14个属性,这些属性会随着消息一起发送)中设置两个值replyTo(一个Queue名称,用于告诉服务器处理完成后将通知我的消息发送到这个Queue中)和correlationId(此次请求的标识号,服务器处理完成后需要将此属性返还,客户端将根据这个id了解哪条请求被成功执行了或执行失败)
2、服务器端收到消息并处理
3、服务器端处理完消息后,将生成一条应答消息到replyTo指定的
Queue
,同时带上correlationId属性
4、客户端之前已订阅replyTo指定的Queue,从中收到服务器的应答消息后,根据其中的correlationId属性分析哪条请求被执行了,根据执行结果进行后续业务处理

(七)、消息确认:Message acknowledgment

在实际应用中,可能会发生消费者收到Queue中的消息,但没有处理完成就宕机(或出现其他意外)的情况,这种情况下就可能会导致消息丢失。为了避免这种情况发生,我们可以要求消费者在消费完消息后发送一个回执给RabbitMQRabbitMQ收到消息回执(Message acknowledgment)后才将该消息从Queue中移除;如果RabbitMQ没有收到回执并检测到消费者的RabbitMQ连接断开,则RabbitMQ会将该消息发送给其他消费者(如果存在多个消费者)进行处理。这里不存在Timeout概念,一个消费者处理消息时间再长也不会导致该消息被发送给其他消费者,除非它的RabbitMQ连接断开
这里会产生另外一个问题,如果我们的开发人员在处理完业务逻辑后,忘记发送回执给RabbitMQ,这将会导致严重的问题,Queue中堆积的消息会越来越多,消费者重启后会重复消费这些消息并重复执行业务逻辑
如果我们采用no-ack的方式进行确认,也就是说,每次Consumer接到数据后,而不管是否处理完成,RabbitMQ会立即把这个Message标记为完成,然后从queue中删除了

(八)、消息持久化:Message durability

如果我们希望即使在RabbitMQ服务重启的情况下,也不会丢失消息,我们可以将QueueMessage都设置为可持久化的(durable),这样可以保证绝大部分情况下我们的RabbitMQ消息不会丢失。但依然解决不了小概率丢失事件的发生(比如RabbitMQ服务器已经接收到生产者的消息,但还没来得及持久化该消息时RabbitMQ服务器就断电了),如果我们需要对这种小概率事件也要管理起来,那么我们要用到事务

(九)、分发机制

我们在应用程序使用消息系统时,一般情况下生产者往队列里插入数据时速度是比较快的,但是消费者消费数据往往涉及到一些业务逻辑处理导致速度跟不上生产者生产数据。因此如果一个生产者对应一个消费者的话,很容易导致很多消息堆积在队列里。这时,就得使用工作队列了。一个队列有多个消费者同时消费数据
工作队列有两种分发数据的方式: 轮询分发(Round-robin)和 公平分发(Fair dispatch

轮询分发:队列给每一个消费者发送数量一样的数据

公平分发: 消费者设置每次从队列中取一条数据,并且消费完后手动应答,继续从队列取下一个数据

1、轮询分发:Round-robin dispatching

如果工作队列中有两个消费者,两个消费者得到的数据量一样的,并不会因为两个消费者处理数据速度不一样使得两个消费者取得不一样数量的数据。但是这种分发方式存在着一些隐患,消费者虽然得到了消息,但是如果消费者没能成功处理业务逻辑,在RabbitMQ中也不存在这条消息。就会出现消息丢失并且业务逻辑没能成功处理的情况

2、公平分发:Fair dispatch

消费者设置每次从队列里取一条数据,并且关闭自动回复机制,每次取完一条数据后,手动回复并继续取下一条数据。与轮询分发不同的是,当每个消费都设置了每次只会从队列取一条数据时,并且关闭自动应答,在每次处理完数据后手动给队列发送确认收到数据。这样队列就会公平给每个消息费者发送数据,消费一条再发第二条,而且可以在管理界面中看到数据是一条条随着消费者消费完从而减少的,并不是一下子全部分发完了。采用公平分发方式就不会出现消息丢失并且业务逻辑没能成功处理的情况

(十)、事务

对事务的支持是AMQP协议的一个重要特性。假设当生产者将一个持久化消息发送给服务器时,因为consume 命令本身没有任何Response返回,所以即使服务器崩溃,没有持久化该消息,生产者也无法获知该消息已经丢失。如果此时使用事务,即通过txSelect()开启一个事务,然后发送消息给服务器,然后通过txCommit() 提交该事务,即可以保证,如果txCommit()提交了,则该消息一定会持久化,如果txCommit() 还未提交即服务器崩溃,则该消息不会服务器接收。当然Rabbit MQ 也提供了txRollback() 命令用于回滚某一个事务

(十一)、Confirm机制

使用事务固然可以保证只有提交的事务,才会被服务器执行。但是这样同时也将客户端与消息服务器同步起来,这背离了消息队列解耦的本质。Rabbit MQ提供了一个更加轻量级的机制来保证生产者可以感知服务器消息是否已被路由到正确的队列中——Confirm。如果设置channelconfirm状态,则通过该channel发送的消息都会被分配一个唯一的ID,然后一旦该消息被正确的路由到匹配的队列中后,服务器会返回给生产者一个Confirm,该Confirm包含该消息的ID,这样生产者就会知道该消息已被正确分发。对于持久化消息,只有该消息被持久化后,才会返回Confirm。Confirm机制的最大优点在于异步,生产者在发送消息以后,即可继续执行其他任务。而服务器返回Confirm后,会触发生产者的回调函数,生产者在回调函数中处理Confirm信息。如果消息服务器发生异常,导致该消息丢失,会返回给生产者一个nack,表示消息已经丢失,这样生产者就可以通过重发消息,保证消息不丢失。Confirm机制在性能上要比事务优越很多。但是Confirm机制,无法进行回滚,就是一旦服务器崩溃,生产者无法得到Confirm信息,生产者其实本身也不知道该消息是否已经被持久化,只有继续重发来保证消息不丢失,但是如果原先已经持久化的消息,并不会被回滚,这样队列中就会存在两条相同的消息,系统需要支持去重

(十二)、Alternate Exchange(代替交换器)

Alternate ExchangeRabbitmq自己扩展的功能,不是AMQP协议定义的
创建Exchange指定该ExchangeAlternate Exchange,发送消息的时候如果Exchange没有成功把消息路由到队列中去,这就会将此消息路由到Alternate Exchange属性指定的Exchange上了。需要在创建Exchange时添加alternate-exchange属性。如果Alternate Exchange也没能成功把消息路由到队列中去,这个消息就会丢失。可以触发publish confirm机制,表示这个消息没有确认

创建交换器时需要指定如下属性
Map<String,Object> argsMap = new HashMap<>();
argsMap.put(“alternate-exchange”,“Alternate Exchange Name”);

(十三)、TTL(生存时间)

RabbitMQ允许您为消息和队列设置TTL(生存时间)。 可以使用可选的队列参数或策略完成(推荐使用后一个选项)。 可以为单个队列,一组队列或单个消息应用消息TTL

设置消息的过期时间
MessageProperties messageProperties = new MessageProperties();           
messageProperties.setExpiration(30000);
 
设置队列中消息的过期时间
在声明一个队列时,可以指定队列中消息的过期时间,需要添加x-message-ttl属性
Map<String, Object> arguments = new HashMap<>();
arguments.put(“x-message-ttl”,30000);                        

如果同时制定了Message TTLQueue TTL,则时间短的生效

(十四)、Queue Length Limit(队列长度限制)

可以设置队列中消息数量的限制,如果测试队列中最多只有5个消息,当第六条消息发送过来的时候,会删除最早的那条消息。队列中永远只有5条消息
使用代码声明含有x-max-lengthx-max-length-bytes属性的队列
Max length(x-max-length) 用来控制队列中消息的数量
如果超出数量,则先到达的消息将会被删除掉

Max length bytes(x-max-length-bytes) 用来控制队列中消息总的大小
如果超过总大小,则最先到达的消息将会被删除,直到总大小不超过x-max-length-byte为止

 
Map<String, Object> arguments = new HashMap<>();
arguments.put(“x-max-length”,3);                     #表示队列中最多存放三条消息
Map<String, Object> arguments = new HashMap<>();
arguments.put(“x-max-length-bytes”,10);              #队列中消息总的空间大小

(十五)、Dead Letter Exchange(死信交换器)

在队列上指定一个Exchange,则在该队列上发生如下情况
1、 消息被拒绝(basic.reject or basic.nack),且requeue=false
2、 消息过期而被删除(TTL
3、 消息数量超过队列最大限制而被删除
4、 消息总大小超过队列最大限制而被删除

就会把该消息转发到指定的这个exchange
需要定义了x-dead-letter-exchange属性,同时也可以指定一个可选的x-dead-letter-routing-key,表示默认的routing-key,如果没有指定,则使用消息原来的routeing-key进行转发

当定义队列时指定了x-dead-letter-exchangex-dead-letter-routing-key视情况而定),并且消费端执行拒绝策略的时候将消息路由到指定的Exchange中去
我们知道还有二种情况会造成消息转发到死信队列
一种是消息过期而被删除,可以使用这个方式使的rabbitmq实现延迟队列的作用。还有一种就是消息数量超过队列最大限制而被删除或者消息总大小超过队列最大限制而被删除

(十六)、priority queue(优先级队列)

声明队列时需要指定x-max-priority属性,并设置一个优先级数值

消息优先级属性
MessageProperties messageProperties = new MessageProperties();                
messageProperties.setPriority(priority);           

如果设置的优先级小于等于队列设置的x-max-priority属性,优先级有效
如果设置的优先级大于队列设置的x-max-priority属性,则优先级失效

创建优先级队列,需要增加x-max-priority参数,指定一个数字。表示最大的优先级,建议优先级设置为1~10之间
发送消息的时候,需要设置priority属性,最好不要超过上面指定的最大的优先级
如果生产端发送很慢,消费者消息很快,则有可能不会严格的按照优先级来进行消费
1、发送的消息的优先级属性小于设置的队列属性x-max-priority值,则按优先级的高低进行消费,数字越高则优先级越高
2、送的消息的优先级属性都大于设置的队列属性x-max-priority值,则设置的优先级失效,按照入队列的顺序进行消费
3、 费端一直进行监听,而发送端一条条的发送消息,优先级属性也会失效

RabbitMQ不能保证消息的严格的顺序消费

(十七)、延迟队列

延迟队列就是进入该队列的消息会被延迟消费的队列。而一般的队列,消息一旦入队了之后就会被消费者马上消费
延迟队列多用于需要延迟工作的场景

最常见的是以下两种场景:

1、 消费
如:用户生成订单之后,需要过一段时间校验订单的支付状态,如果订单仍未支付则需要及时地关闭订单
用户注册成功之后,需要过一段时间比如一周后校验用户的使用情况,如果发现用户活跃度较低,则发送邮件或者短信来提醒用户使用
2、延迟重试
如:消费者从队列里消费消息时失败了,但是想要延迟一段时间后自动重试
我们可以利用
RabbitMQ
的两个特性,一个是
Time-To-Live Extensions
,另一个是
Dead Letter
Exchanges。实现延迟队列

Time-To-Live Extensions
RabbitMQ允许我们为消息或者队列设置TTLtime to live),也就是过期时间。TTL表明了一条消息可在队列中存活的最大时间,单位为毫秒。也就是说,当某条消息被设置了TTL或者当某条消息进入了设置了TTL的队列时,这条消息会在经过TTL秒后“死亡”,成为Dead Letter。如果既配置了消息的TTL,又配置了队列的TTL,那么较小的那个值会被取用

Dead Letter Exchange
刚才提到了,被设置了TTL的消息在过期后会成为Dead Letter。其实在RabbitMQ中,一共有三种消息的“死亡”形式:
1、消息被拒绝。通过调用basic.reject或者basic.nack并且设置的requeue参数为false
2、消息因为设置了TTL而过期
3、消息进入了一条已经达到最大长度的队列
如果队列设置了
Dead Letter Exchange(DLX)
,那么这些Dead Letter就会被重新publishDead Letter Exchange,通过Dead Letter Exchange路由到其他队列

四、RabbitMQ配置

(一)、RabbitMQ 常用命令

1、Vhost虚拟机

每个RabbitMQ服务器都能创建虚拟主机(virtual host),简称vhost。每个vhost本质上是一个独立的小型RabbitMQ服务器,拥有自己独立的队列、交换器及绑定关系等,并且它拥有自己独立的权限,RabbitMQ默认创建vhost/

 
创建vhost
rabbitmqctl add_vhost {
    
    vhost}                           
 
查看所有vhost
rabbitmqctl list_vhosts                   
 
删除指定vhost
rabbitmqctl delete_vhost {
    
    vhost}                    

2、用户管理

RabbitMQ中,用户是访问控制的基本单元,且单个用户可以跨越多个vhost进行授权

 
创建用户
rabbitmqctl add_user {
    
    username} {
    
    password}                   
 
修改密码
rabbitmqctl change_password {
    
    username} {
    
    password}                  
 
清除密码
rabbitmqctl clear_password {
    
    username}                         
 
验证用户
rabbitmqctl authenticate_user {
    
    username} {
    
    password}                
 
删除用户
rabbitmqctl delete_user {
    
    username}                  
 
用户列表
rabbitmqctl list_users          

3、权限管理

用户权限指的是用户对exchangequeue的操作权限,包括配置权限,读写权限。配置权限会影响到exchangequeue的声明和删除。读写权限影响到从queue里取消息,向exchange发送消息以及queueexchange的绑定(bind)操作

RabbitMQ中,权限控制是以vhost为单位,创建用户时,将被指定至少一个vhost,默认的vhost“ / ”

 
授予权限
rabbitmqctl set_permissions [-p vhost] {
    
    user}{
    
    conf}{
    
    write}{
    
    read}            
配置项 说明
vhost 授权用户访问指定的vhost
user 用户名
conf 一个用于匹配用户在哪些资源上拥有可配置权限的正则表达式,例如:".*"表示全部
write 一个用于匹配用户在哪些资源上拥有可写权限的正则表达式 ,例如:".*"表示全部
read 一个用于匹配用户在哪些资源上拥有可读权限的正则表达式,例如:".*"表示全部
 
收回权限
rabbitmqctl clear_permissions [-p vhost] {username}              
 
虚拟主机权限列表
rabbitmqctl list_permissions [-p vhost]                           
 
查看指定用户权限
rabbitmqctl list_user_permissions {username}            

4、角色分配

rabbitmq的角色有5种类型

 
rabbitmqctl set_user_tags {
    
    username} {
    
    tag…}            
User为用户名
Tag为角色名(对应的administrator,monitoring,policymaker,management,或其他自定义名称)
 
也可以给同一用户设置多个角色,例:
rabbitmqctl  set_user_tags  hncscwc  monitoring  policymaker                 
配置项 说明
none其他 无任何角色,新创建的用户默认角色为none
management普通管理者 可以访问web管理页面,无法看到节点信息,也无法对策略进行管理
policymaker策略制定者 包含management的所有权限,并可以管理策略和参数,但无法查看节点的相关信息
monitoring监控者 包含management的所有权限,并可以看到所有连接(启用management plugin的情况下)、信道及节点相关信息(进程数,内存使用情况,磁盘使用情况等)
administartor超级管理员 包含minitoring的所有权限,并可以管理用户、虚拟主机、权限、策略、参数

5、Web端管理

RabbitMQ Management 插件可以提供Web界面来管理RabbitMQ中的虚拟主机、用户、角色、队列、交换器、绑定关系、策略、参数等,也可用于监控RabbitMQ服务的状态及一些统计信息

 
启动插件
rabbitmq-plugins enable rabbitmq_management                         
 
关闭插件
rabbitmq-plugins disable rabbitmq_management                           
 
插件列表:其中标记为[E*]为显示启动,其中标记为[e*]为隐式启动,开启此功能后需要重启服务才可以正式生效
rabbitmq-plugins list                         
 
 

6、RabbitMQ 管理

 
[root@servers sbin]# ./rabbitmq-server                  #启动,ctrl + c 退出及关闭
 
[root@servers sbin]# ./rabbitmq-server -detached        #在后台启动服务
 
[root@servers sbin]# ./rabbitmqctl stop                 #关闭服务
 
[root@C7--13 sbin]# ./rabbitmq-server status            #查看状态

7、查看队列列表

rabbitmqctl list_queues[-p vhost][queueinfoitem…]            
返回列 说明
name 队列名称
durable 队列是否持久化
auto_delete 队列是否自动删除
arguments 队列参数
policy 应用到队列上的策略名称
pid 队列关联的进程ID
owner_pid 处理排他队列连接的进程ID
exclusive 队列是否排他

8、查看交换器列表

rabbitmqctl list_exchanges [-p vhost][exchangeinfoitem…]
返回列 说明
name 交换器名称
type 交换器类型
durable 交换器是否持久化
auto_delete 交换器是否自动删除
internal 是否是内置交换器
arguments 交换器的参数
policy 交换器的策略

9、查看绑定关系的列表

rabbitmqctl list_bindings [-p] [bindinginfoitem…]             
返回列 说明
source_name 消息来源的名称
source_kind 消息来源的类别
destination_name 消息目的地的名称
destination_kind 消息目的地的种类
routing_key 绑定的路由键
arguments 绑定的参数

10、查看连接信息列表

rabbitmqctl list_connections [connectioninfoitem …]            
返回列 说明
pid 与连接相关的进程ID
name 连接名称
port 服务器端口
host 服务器主机名
peer_port 服务器对端端口。当一个客户端与服务器连接时,这个客户端的端口就是peer_port
peer_host 服务器对端主机名称,或IP
ssl 是否启用SSL
state 连接状态,包括starting\tning\opening\running\flow\blocking\blocked\closing\closed
channels 连接中的信道个数
protocol 使用的AMQP协议版本
user 与连接相关的用户名
vhost 与连接相关的vhost名称
timeout 连接超时时长,单位秒

11、查看信道列表

rabbitmqctl list_channels [channelinfoitem…]                
返回列 说明
pid 与连接相关的进程ID
connection 信道所属连接的进程ID
name 信道名称
number 信道的序号
user 与连接相关的用户名
vhost 与连接相关的vhost名称
transactional 信道是否处于事务模式
confirm 信道是否处于 publisher confirm模式
consumer_count 信道中的消费者个数
messages_unacknowledged 已投递但是还未被ack的消息个数
messages_uncommitted 已接收但是还未提交事务的消息个数
acks_uncommitted ack收到但是还未提交事务的消息个数
messages_unconfirmed 已发送但是还未确认的消息个数
perfetch_count 消费者的Qos个数限制,0表示无上限
global_prefetch_count 整个信道的Qos个数限制

12、查看消费者列表

rabbitmqctl list_consumers [-p vhost]            
返回列 说明
arguments 参数
channel_pid Channel process id
consumer_tag consumer mark
prefetch_count The limit on the number of Qos for consumers , 0 means no upper limit
queue_name queue name

(2) Basic operations of web interface

1. Add user

img

img

2. Create a virtual host img3. Bind users to the virtual host

img

img

img

Guess you like

Origin blog.csdn.net/qq_36306519/article/details/131234077