concept
What is the message
- It refers to message data transfer between the two separate systems. Both systems can be two computers, it can also be two processes.
- Messages are platform-independent and language-independent!
What is the queue
- Queue is a data structure, an array or linked list internally implemented,
- Characteristics of the tail of the queue is placed only, queue head removed, i.e. First In First Out [FIFO]
- The operation of the queue have the team and the team
is, you have a program-generated content and then into the team (producer)
another program reads the content, the content team (consumers)
What is the message queue
- This simply means that: the queue as containers used to hold messages during transmission message.
Queue channel is in the transmission message, the message is stored in the container,
depending on the situation, there may be FIFO, the difference between priority queues.
Why do you want to use the message queue
Decoupling
Message Queuing can decouple the business logic, the caller only needs to give orders rather than waiting for the entire logic is finished!
For example: when you need to call registered three services, these three services can be independently placed on three servers, the implementation of which step to send a message directly to asynchronous calls. Registered efficiency is much faster
to call the mail service: send a registration message with a verification link,
call the third-party verification service: verify the true and false identity information,
call the user's service: the user registration.
Synchronous asynchronous transfer
Synchronizing process can be processed into asynchronous
writes messages to the message queue, the non-essential business logic operates in an asynchronous manner, response speed
order is finished directly returned to the user a result, only takes 50ms, and then notifies MQ do follow-up things.
Clipping
In a highly concurrent scenarios smooth short span of time [] service request
shunt: a large burst request into the request queue can withstand the rear end.
When using message queues it
Follow execution result downstream, with RPC / REST
not concerned with the execution result downstream, with the MQ, without RPC / REST
sensitive, is better than the RPC message queue to ensure that the transaction and a strong need for a delay.
For example:
your server can handle 100 orders for one second, one second spike activity but come 1000 orders for 10 seconds, in the case of the back-end capacity can not be increased,
you can use the message queue A total of 10,000 requests in the queue pressure where, according to the original consumer the ability to handle the background, after 100 seconds finished processing all the requests (rather than directly down the loss of order data).
note
mq concern is the "notice" rather than "deal
Simply put: MQ messages can only be guaranteed in accordance with the order to inform the consumer, can not guarantee consumer processing logic, for example: the order is not executed.
Suppose there are three messages: M1 (text messaging), M2 (e-mail), M3 (push station)
the order in the queue are: M3, M2, M1 MQ ensure message when consumption is in this order,
but can not guarantee consumer, you must first send push the station, and then send e-mail, text messaging and finally,
because the three consumer business hours received message execution is likely not the same.
Installation Rabbit MQ
Installation ErLang
Erlang ([ 'ə: læŋ]) is a general purpose-oriented concurrent programming language, which consists of Swedish telecom equipment manufacturer Ericsson under the jurisdiction of CS-Lab development, it aims to create a programming language that can cope with large-scale concurrent activities and operating environment.
rpm --import https://packages.erlang-solutions.com/rpm/erlang_solutions.asc
vi /etc/yum.repos.d/xxx (xxx is an arbitrary directory of an existing list of files yum)
increase in the content of the following documents:
[erlang-solutions]
name=Centos $releasever - $basearch - Erlang Solutions
baseurl=https://packages.erlang-solutions.com/rpm/centos/$releasever/$basearch
gpgcheck=1
gpgkey=https://packages.erlang-solutions.com/rpm/erlang_solutions.asc
enabled=1Yum cache information generated
yum makecache
Mounting the ErLang
yum the install Erlang -Y
Check the installation results, see ErLang version
erl -version
Installation Rabbit Mq
wget https://www.rabbitmq.com/releases/rabbitmq-server/v3.6.6/rabbitmq-server-3.6.6-1.el6.noarch.rpm --no-check-certificate
rpm --import https://www.rabbitmq.com/rabbitmq-release-signing-key.asc
yum -y install rabbitmq-server-3.6.6-1.el6.noarch.rpm
Reference can be given:
mounting Rabbit MQ
Start Rabbit MQ
Daemon is configured to start automatically with the system, execute with root privileges:
chkconfig RabbitMQ Server-ON
start RabbitMQ service
service rabbitmq-server start
RabbitMQ service status checking
service rabbitmq-server status
RabbitMQ installation of the WEB management interface
rabbitmq-plugins enable rabbitmq_management
RabbitMQ user settings and permissions granted
Create an account
rabbitmqctl add_user test 123456
Set User Role
rabbitmqctl set_user_tags test administrator
to set user privileges
rabbitmqctl set_permissions -p "/" the Test. " ". " ". "*"
After the setup is complete you can view the current users and roles (need to open the service)
rabbitmqctl list_users
WEB browser to access the management interface
HTTP: // RabbitMQ-Server-ip: 15672
RabbitMQ RabbitMQ-Server-ip is the host physical machine in accordance with the IP.
WEB management interface port RabbitMQ provides for 15672
RabbitMQ principle
Schematic
Message
There are two parts: Header and Body.
Header is a set various attributes on Producer added,
these properties can be controlled Message whether the cache, which received queue, priority is much like.
Body is a real need to send data, it is not visible Broker binary data stream, should not be affected during transmission.
(In rabbitMQ, the message may be stored in any type of java object, must implement serialization (Serializable))
Producer Publisher message
The client application is also a published message to the switch
Consumers Consumer news
It represents a client application to obtain the message from the message queue.
Exchange switch.
For receiving a message sent by the producer of the queue and the server to route the messages.
Three common types of switches
direct (publish and subscribe exact match)
fanout (broadcast)
Topic (themes, rules match)
Routing Routing-key key
RabbitMQ decided to deliver the message to which the rule queue.
Bound to the queue via the switch routing keys.
MQ message is sent to the server, the message will have a routing key, even if empty, RabbitMQ routing keys which will also be used for matching and binding.
If they match, the message will be delivered to the queue.
If not, the message will enter a black hole.
Binding Binding
For the association between a message queue [] and [] switch. Binding is a key-based routing rule and the switch connecting the message queue, it can be appreciated that the switch into a routing table constituted by the binding.
Queue message queue.
To save messages until sent to the consumer.
It is a container for the message, the message is the end. A message can be put into one or more queues.
Message has been in the queue inside, waiting for consumers to link to this queue will remove it.
Connection
It refers to the TCP link rabbit servers and services established.
Channel
Channel, which is a virtual TCP link. A TCP may create multiple channels on a connection.
TCP Once opened, it will create AMQP channels. Whether release message, receiving a message, the subscription queue, these actions are accomplished via a channel
Virtual Host
It represents a set of switches, message queues, and related objects.
It is essentially a mini version of RabbitMQ server, with its own queue, switches, binding authority and a mechanism vhost.
Similarly there are a mysql as N databases.
Borker
It represents the message queue server entity. RabbitMQ is the overall application.
Relationship between the exchanger and the queue
Switch key and by routing queues bound together, if the message has a routing key with keys queues and routing switches match, then the message will be routed to the queue binding.
That is, the process message queue, the message will go through the first switch, the next routing switch in the distribution key by matching to a specific message queue.
Routing rules can be understood as the key match.
Why do we need RabbitMQ channel? Why not direct TCP communication?
Creation and destruction overhead of TCP particularly large.
The need to create 3-way handshake, the need to destroy four times to break up.
If you do not believe, that is what the application will link to TCP Rabbit, when hundreds of thousands of messages per second peak link will cause a huge waste of resources, and the operating system per second TCP link count is limited, will cause performance bottlenecks.
Principle is a thread channel of a channel, multi-threads multiple channels with the use of a TCP connection. A TCP connection can accommodate unlimited channel, even thousands of requests per second does not become a performance bottleneck.
The general process
consumer registration queue listener to Broker (RabbitMQ)
Consumer first register a listener queue to monitor the state of the queue, the queue status changes when news consumption,
the time of registration queue listener needs to provide:
- Exchange (switch) information:
exchange type (Dircet direct, Topic topic, Fanout broadcast), switch name, whether to automatically delete - Queue (queue) information,
name, whether to automatically delete - And Routing Key (routing key) information.
Since a key defined value, this value is Queue Exchange and the connection identification.
producer sends a message to queue
Send a message to the producer RabbitMQ, specify the Exchange (switch) the information in the message header, Routing Key (routing key) information
Broker (RabbitMQ) match
Found RebbitMQ exchanger by Producer specified Exchange name and find the corresponding queue by specifying the Routing key, the message is queued.
Queue status changes, Consumer listener will get the message through and consumption.
consumer how to do a cluster of news consumption
Suppose I send a text message service, in order to ensure the stability of SMS, SMS made a cluster service, this time MQ message is how to be consumed.
Exchange
Its role: to receive messages sent by the producer and to route the messages to the server queues.
Exchange is matched by the corresponding Queue Routing Key.
We need to know the type and RabbitMQ in Exchange Queue, as well as Routing key is provided by the end consumer,
producer and only provides Exchange Routing key, broker finds the corresponding switch in accordance with Exchange to provide the name of producer, and then
according to the routing key to match the corresponding queue, placed in the message queue.
There are several types of Exchange:
Routing Key Direct Exchange is the type of the whole match.
Routing key Topic type of Exchange is part of matching or fuzzy matching.
Routing key Fanout type of Exchange is to give up the match.
In the same match are certainly a limit in Exchange, Exchange is the same match.
Reliability message processing
News endurance of
To ensure that messages are not lost in the MQ.
Case of message loss
- consumer does not start, and the producer sent a message, the message will be lost.
- When all of the consumer downtime, queue will auto-delete, the message will still be lost
Message acknowledgment.
necessity
consumer receive a message in the process of consumption program to exhibit unusual or network interruption, if not ack then, MQ put the message deleted, causing data loss.
process
RabbitMQ The news pushed to the Consumer, RabbitMQ will break the news to lock in the locked state of the message will not be repeated push is secondary consumer.
Other consumer can continue to spend the next message, send an ack to RabbitMQ, RabbitMQ will be deleted when the news consumer news consumer after the completion of confirmation.
If more than a certain time RabbitMQ consumer does not receive the ack, the news will be unlocked, back into the head of the queue, ensure the order of the messages.
Memory leak may
If the Consumer does not deal with message confirmation, it will lead to serious consequences.
Assuming that all of the Consumer did not confirm the normal feedback information, and quit listening state, then the message is stored permanently, and is locked up until the message is normal consumption.
The Producer messages continue to continue to send a message to RabbitMQ, then the message will accumulate for memory for the server where the RabbitMQ, result in a "memory leak" problem.
solution:
- Configuration message retries.
By global configuration file, open the message retry mechanism consumption, configure the number of retries.
When RabbitMQ not received confirmation of Consumer feedback will decide the number of retries push message, when the number of retries after use, regardless of whether it has received acknowledgment feedback, RabbitMQ will delete messages, to avoid possible memory leaks depending on the configuration.
At the end consumer specific configuration is as follows:
#开启重试
spring.rabbitmq.listener.retry.enabled=true
#重试次数，默认为3次
spring.rabbitmq.listener.retry.max-attempts=5- Exception handling encoding
mode exception handling by encoding, to ensure the normal execution message confirmation mechanism.
Such as: catch block, the success of the unprocessed message to resend MQ.
Such as: catch code, the local retry logic (using the timing thread pool tasks repeated 3 times.)
As: catch code, the message is stored in the DB exception, then the timing task to clear the message.
Repeated consumption
- RabbitMQ waiting assume ack timeout is 1s, and consumer news consumer needs 2s, then this message appears ack wait timeout, back into the queue, which appeared repeated consumption.
- After receiving news consumer interrupt the Connection, the message will be re-placed in the queue, there will be repeated consumption.
- Suppose the consumer end of message processing systems occur when the abnormality can not send acknowledgment mechanism.
【Solution】
- Duration of the execution of a test consumer, and defining a reasonable length of time MQ ack timeout.
- Add as a message version or timestamp, or be re-sentenced based on business id.
If you can not duplicate is not mandatory consumption, it is best not to judge.
RabbitMQ message acknowledgment is enabled by default, it does not recommend closed.
Direct Switch
It is the point (point to point) to achieve [publish / subscribe] standard switch. Here exchanger is (Exchange).
Business scene
producer side code implemented
pom-dependent
Inheritance spring-boot-starter-parent
introduced RabbitMQ: the Spring AMQP-the Boot-Starter-
RabbitMQ dependent. rabbitmq has been done to integrate spring-boot access implementation.
spring cloud also springboot made integration logic. Therefore rabbitmq dependent used directly in the spring cloud.
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.bjsxt</groupId>
<artifactId>rabbitmq-direct-producer</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>rabbitmq-direct-producer</name>
<description>Demo project for Spring Boot</description>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>1.5.13.RELEASE</version>
<relativePath /> <!-- lookup parent from repository -->
</parent>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<java.version>1.8</java.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<!-- rabbitMQ的依赖。rabbitmq已经被spring-boot做了整合访问实现。
spring cloud也对springboot做了整合逻辑。所以rabbitmq的依赖可以在spring cloud中直接使用。
-->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-amqp</artifactId>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
</project>
Configuration RabbitMQ
spring.application.name=direct-producer
server.port=8082
# 必要配置
# 配置rabbitmq链接相关信息。key都是固定的。是springboot要求的。
# rabbitmq安装位置
spring.rabbitmq.host=192.168.1.122
# rabbitmq的端口
spring.rabbitmq.port=5672
# rabbitmq的用户名
spring.rabbitmq.username=test
# rabbitmq的用户密码
spring.rabbitmq.password=123456Create a message carrier object
- Object must implement the serialization interface.
Here the getter and setter methods omitted.
/**
* 消息内容载体，在rabbitmq中，存储的消息可以是任意的java类型的对象。
* 强制要求，作为消息数据载体的类型，必须是Serializable的。
* 如果消息数据载体类型未实现Serializable，在收发消息的时候，都会有异常发生。
*/
public class LogMessage implements Serializable {
private Long id;
private String msg;
private String logLevel;
private String serviceType;
private Date createTime;
private Long userId;
public LogMessage() {
super();
}
public LogMessage(Long id, String msg, String logLevel, String serviceType, Date createTime, Long userId) {
super();
this.id = id;
this.msg = msg;
this.logLevel = logLevel;
this.serviceType = serviceType;
this.createTime = createTime;
this.userId = userId;
}
@Override
public String toString() {
return "LogMessage [id=" + id + ", msg=" + msg + ", logLevel=" + logLevel + ", serviceType=" + serviceType
+ ", createTime=" + createTime + ", userId=" + userId + "]";
}
}Write test classes
Use spring boot [AmqpTemplate provide default implementations RabbitMQ Interface [R] RabbitTemplate object sends a message.
Wherein convertAndSend method may send a message:
This method is the incoming normal java object, converted to a message type in a subject in need rabbitmq, and sends the message to the rabbitmq.
A parameter: the name of the switch. Type String
Parameter Two: routing keys. Type String
parameter three: the message is the message you want to send the content object. Type Object
/**
* Direct交换器
* Producer测试。
* 注意：
* 在rabbitmq中，consumer都是listener监听模式消费消息的。
* 一般来说，在开发的时候，都是先启动consumer，确定有什么exchange、queue、routing-key。
* 然后再启动producer发送消息。
*/
@RunWith(SpringRunner.class)
@SpringBootTest(classes=SpringbootServerApplication.class)
public class QueueTest {
@Autowired
private AmqpTemplate rabbitAmqpTemplate;
/*
* 测试消息队列
*/
@Test
public void testSendInfo()throws Exception{
Long id = 1L;
while(true){
Thread.sleep(1000);
final LogMessage logMessage = new LogMessage(id, "test log", "info", "订单服务", new Date(), id);
this.rabbitAmqpTemplate.convertAndSend("log.direct", "log.error.routing.key", logMessage);
id++;
}
}
/*
* 测试消息队列
*/
@Test
public void testSendError()throws Exception{
Long id = 1L;
while(true){
Thread.sleep(1000);
final LogMessage logMessage = new LogMessage(id, "test log", "info", "订单服务", new Date(), id);
this.rabbitAmqpTemplate.convertAndSend("log.direct", "log.info.routing.key", logMessage);
id++;
}
}
}Implement consumer side
pom
And end the same producer
Write info level log consumer code
@Component
@RabbitListener(
bindings=@QueueBinding(
value=@Queue(value="log.error",autoDelete="false"),
exchange=@Exchange(value="log.direct",type=ExchangeTypes.DIRECT),
key="log.error.routing.key"
)
)
public class ErrorReceiver {
/**
* 消费消息的方法。采用消息队列监听机制
* @RabbitHandler - 代表当前方法是监听队列状态的方法，就是队列状态发生变化后，执行的消费消息的方法。
* 方法参数。就是处理的消息的数据载体类型。
*/
@RabbitHandler
public void process(LogMessage msg){
System.out.println("Error..........receiver: "+msg);
}
}@RabbitListener
可以注解类和方法，
注解类：当表当前类的对象是一个rabbit listener。监听逻辑明确，可以由更好的方法定义规范。 必须配合@RabbitHandler才能实现rabbit消息消费能力。
注解方法：代表当前方法是一个rabbit listener处理逻辑。方便开发，一个类中可以定义若干个listener逻辑。方法定义规范可能不合理。
代表当前类型是一个rabbitmq的监听器。
bindings:绑定队列@QueueBinding
@RabbitListener.bindings属性的类型。绑定一个队列。
value:绑定队列， Queue类型。
exchange:配置交换器， Exchange类型。
key:路由键，字符串类型。@Queue - queue.
value:队列名称
autoDelete:是否是一个临时队列(也就是所有的consumer关闭后是否删除队列)
true ： 删除
false：如果queue中有消息未消费，无论是否有consumer，都保存queue。@Exchange - switch
value:为交换器起个名称 type:指定具体的交换器类型
@RabbitHandler
代表当前方法是监听队列状态的方法，就是队列状态发生变化后，执行的消费消息的方法。Error writing the code level log consumption
@Component
@RabbitListener(
bindings=@QueueBinding(
value=@Queue(value="log.info",autoDelete="false"),
exchange=@Exchange(value="log.direct",type=ExchangeTypes.DIRECT),
key="log.info.routing.key"
)
)
public class InfoReceiver {
@RabbitHandler
public void process(LogMessage msg){
System.out.println("Info........receiver: "+msg);
}
}Topic switch
Scenes
There are customer service, order services, goods and services three services, each service will have logs, logs are divided info, error and other levels can be achieved using MQ to collect logs.
Use Direct switch, you need to define at least six queues.
If Topic exchanger may simplify the development of the consumer side:
achieve
- pom dependency the same as above.
The main types of modified end consumer Exchange Routing key and the corresponding rules
consumer end
Error log handling consumer
@Component
@RabbitListener(
bindings=@QueueBinding(
value=@Queue(value="${mq.config.queue.error}",autoDelete="true"),
exchange=@Exchange(value="${mq.config.exchange}",type=ExchangeTypes.TOPIC),
key="*.log.error"
)
)
public class ErrorReceiver {
@RabbitHandler
public void process(String msg){
System.out.println("......Error........receiver: "+msg);
}
}Log handling consumer Info
@Component
@RabbitListener(
bindings=@QueueBinding(
value=@Queue(value="${mq.config.queue.info}",autoDelete="true"),
exchange=@Exchange(value="${mq.config.exchange}",type=ExchangeTypes.TOPIC),
key="*.log.info"
)
)
public class InfoReceiver {
@RabbitHandler
public void process(String msg){
System.out.println("......Info........receiver: "+msg);
}
}producer end
Commodity log information
@Component
public class ProductSender {
@Autowired
private AmqpTemplate rabbitAmqpTemplate;
/*
* 发送消息的方法
*/
public void send(String msg){
//向消息队列发送消息
//参数一：交换器名称。
//参数二：路由键
//参数三：消息
this.rabbitAmqpTemplate.convertAndSend("log.topic","product.log.info", "product.log.info....."+msg);
this.rabbitAmqpTemplate.convertAndSend("log.topic","product.log.error", "product.log.error....."+msg);
}
}Users send a message
@Component
public class UserSender {
@Autowired
private AmqpTemplate rabbitAmqpTemplate;
/*
* 发送消息的方法
*/
public void send(String msg){
//向消息队列发送消息
//参数一：交换器名称。
//参数二：路由键
//参数三：消息
this.rabbitAmqpTemplate.convertAndSend("log.topic","user.log.info", "user.log.info....."+msg);
this.rabbitAmqpTemplate.convertAndSend("log.topic","user.log.error", "user.log.error....."+msg);
}
}Like Order code omitted.
Fanout Switch
This is simpler, you do not need to be directly arranged on the end consumer and producer Routing key on the list.