Messaging middleware --RabbitMQ (seven) advanced features all here! (On)

Foreword

In front of us the RabbitMQ installation, comparing all major messaging middleware, AMQP core concepts, using the control station, quick start RabbitMQ. This chapter describes the advanced features of RabbitMQ. Two points (upper / lower) are introduced.

How message delivery guarantee 100% success?
Detailed concept Idempotency
In the peak of the business generated by the massive orders, how to avoid the problem of duplication of spending a message?
Confirm acknowledgment message, Return a return message

How to guarantee 100% 1 message delivery success?

1.1 What is the reliability of the delivery end of the production?

Ensure the success of the message sent
MQ nodes successfully receive protection
MQ sender receives node (Broker) acknowledgment
Improve the compensation mechanism for message

The first three steps may not be able to guarantee delivery of messages can be 100% successful. So to add a fourth step

BAT / TMD maker of Internet solutions: - Message for falling, marking the message status when sending a message, the message needs to be persisted to the database, and give this message to set up a state (unsent, transmission, Arrivals). When the message status has changed, the message needs to make a change. Do operations on a message does not arrive in rotation and re-sent. The number of times in rotation also needs to be done 3-5 times a limit. Ensure that messages can be sent successfully.

Delayed delivery of the message, do secondary confirmation, check the callback

Which specific embodiment employed, but also depends on the amount of concurrency and message traffic.

1.2 The first embodiment:

Production end - Reliability Delivery

Graphic:

Blue section, said: producer is responsible for sending messages to the Broker end Biz DB: order database MSG DB: message data face of small-scale applications plus transaction methods can be used to ensure the consistency of the transaction. However, manufacturers in the face of high concurrency, and did not increase the transaction, the transaction splicing performance is very serious, but do compensation.

For example: the following orders send a message.

step1: storage order message (creating an order), business data storage, storage news. Disadvantages: persistence twice. (Status: 0) step2: Under the premise of a successful step1 send a message step3: Broker after receipt of the message, confirm to our production end. Confirm Listener listens for messages Broker asynchronous postback. step4: gripping a specified message, update (status = 1), indicates that the message has been delivered successfully.

step5: Distributed Task Timing acquisition status message, if it is equal to 0 to fetch data. step6: Resend Message step7: retry limit is set three times. If the message retry three times or failed, then (status = 2), we believe the news is a failure.

Why query these messages fails, you may need to manually query.

Suppose step2 executed successfully, step3 due to network glitches. It will never receive confirm the message, then we need to set a rule: For example: When a message warehousing, set a threshold timeout = 5min, after more than 5min, this data will crawl out. A task or write timing of every five minutes, the message status = 0 will crawl out. There may be a small problem: the message sent out regular tasks and just just executed, Confirm not yet received, will perform regular tasks, will lead to the implementation of the message twice. Finer operation: Message Timeout tolerance limits. confirm message is not received within 2-3 minutes, then re-transmitted.

We guarantee MQ think if the first delivery reliability, is appropriate in a highly concurrent scenarios?

The first embodiment has two data storage, a data storage service, a message storage. Such storage of data is a bottleneck. In fact, we only need to be put in storage business.

Delayed delivery of the message, do secondary confirmation, check the callback

This approach does not necessarily guarantee 100 percent success, but also to ensure the success of 99.99% of the news. If you encounter a particularly extreme case, then you can only required labor to compensate for, or to do regular tasks. The second way is mainly to reduce the operation of the database.

The second way to look at:

Graphic:

Upstream service: the production side DownStream service: consumer end Callback service: Callback Service

step1: After successful storage business news, the first message sent. step2: the same message after successful storage, sending a second message, these two messages are simultaneously transmitted. The second message is a delay check, you may be provided 2min, 5min transmission delay. step3: the consumer side listening on the specified queue. step4: After the end consumer handling the messages, a new internally generated message send confirm. Post to MQ Broker. step5: Callback Service Callback Service listens MQ Broker, if you receive a message sent Downstream service, you can send a message to determine the success of the implementation of the message store to MSG DB. step6: Check Detail Check the monitor step2 delayed delivery of a message. At this point two is not the same queue listener, after 5 minutes, Callback service received message, check MSG DB. If it is found before the message has been delivered successfully, you do not need to do other things. If the check fails found, the Callback compensating sends RPC communication. Notifies the upstream end of the production resend the message.

The aim: to do less a DB storage. Concerns not a hundred percent successful delivery, but performance.

2. Idempotence concept

2.1 What sex is power and so on?

Idempotent (idempotent, idempotence) is a mathematical and computer science concept common abstract algebra, i.e., f (f (x)) = f (x). It is simply a result of the operation performed multiple times to produce results consistent with the first performance generated .

We can learn from optimistic locking mechanism of the database:
For example, we execute a SQL statement to update the inventory:
UPDATE T_REPS SET COUNT = COUNT - 1,VERSION = VERSION + 1 WHERE VERSION = 1

Using the plus version Version ways to ensure idempotency.

2.2 Consumer end - idempotency guarantee

In the peak of the business generated by the massive orders, how to avoid duplication of spending problem message?

In the case of high concurrency, there will be plenty of news to reach the MQ, the consumer side needs to listen to a lot of news. In such a case, it is inevitable to repeat delivery message, the network will flash and the like. If you do not idempotent, repeated consumption message appears. - Consumer-side implementation idempotency, it means that our message will never be consumed several times, even though we have received the same message multiple, it will only be performed once.

Look at the mainstream of the Internet giant idempotent operations: - Unique ID + fingerprint right mechanism to re-use database primary key. - Redis atomic implemented using - the other technology idempotent

2.2.1 The only ID + fingerprint code mechanism

The only ID + fingerprint right mechanism to re-use database primary key. Only guarantee
SELECT COUNT (1) FROM T_ORDER WHERE ID = ID + fingerprint unique code if no query is added. There is no need to do anything, the consumer side does not need to consume messages.
Benefits: simple
Disadvantages: There are performance bottlenecks database write under high concurrency
Solution: Follow-up ID is divided routing algorithm library sub-table-sharing traffic pressure.

2.2.2 Redis atom property

The simplest use Redis self-energizing.

Use Redis be idempotent, issues to be considered.
First: Do we need the data off the library, if the library is off, then the key to solving the problem is how to do the database and cache atomicity? Plus a transaction not, and Redis database transaction is not the same, can not guarantee success while at the same time fail. You have any better options?
Second: If not for falling, so are stored in the cache, how to set up strategies timing synchronization? How to achieve stability cached data?

3. Confirm the confirmation message

Confirm understanding of message acknowledgment mechanism:

Confirmation message refers to the producer after the delivery of the message, if Broker message is received, the producer will give us a response.
Producers receive a response, to determine whether the message is sent to the normal Broker, the core of this approach is to protect the reliability of the delivery of the message!

Confirm acknowledgment message flow diagram

Blue: producer producer red: MQ Broker server

Producers send messages to end Broker, Broker back to the producer after the receipt of the message. Confirm Listener listener response.

Operations are asynchronous, when the producer after sending message, do not need the tube. Confirm Listener listens MQ Broker response.

3.1 How to achieve Confirm confirmation message?

Step: open channel on Acknowledged Mode: channel.confirmSelect () a second step; added listen on Chanel: addConfirmListener , success and failure monitor returns the result of the re-transmitted message depending on the result of logging, or the like follow-up treatment!

3.2 coding:

Producer:


/**
 * 
* @ClassName: Producer 
* @Description: 生产者
* @author Coder编程
* @date 2019年7月30日 上午21:27:02 
*
 */
public class Producer {

	
	public static void main(String[] args) throws Exception {
		
		
		//1 创建ConnectionFactory
		Connection connection = ConnectionUtils.getConnection();
	
		//2 通过Connection创建一个新的Channel
		Channel channel = connection.createChannel();
		
		
		//3 指定我们的消息投递模式: 消息的确认模式 
		channel.confirmSelect();
		
		String exchangeName = "test_confirm_exchange";
		String routingKey = "confirm.save";
		
		//4 发送一条消息
		String msg = "Hello RabbitMQ Send confirm message!";
		channel.basicPublish(exchangeName, routingKey, null, msg.getBytes());
		
		//5 添加一个确认监听  用于发送消息到Broker端之后，回送消息的监听
		channel.addConfirmListener(new ConfirmListener() {
			@Override
			public void handleNack(long deliveryTag, boolean multiple) throws IOException {
				System.err.println("-------no ack!-----------");
			}
			
			@Override
			public void handleAck(long deliveryTag, boolean multiple) throws IOException {
				System.err.println("-------ack!-----------");
			}
		});
	}
}



复制代码

consumer:


/**
 * 
* @ClassName: Consumer 
* @Description: 消费者
* @author Coder编程
* @date 2019年7月30日 上午21:32:02 
*
 */
public class Consumer {

	
	public static void main(String[] args) throws Exception {
		
		
		//1 获取一个连接 
        Connection connection = ConnectionUtils.getConnection();
		
		//2通过Connection创建一个新的Channel
		Channel channel = connection.createChannel();
		
		String exchangeName = "test_confirm_exchange";
		String routingKey = "confirm.#";
		String queueName = "test_confirm_queue";
		
		//3 声明交换机和队列 然后进行绑定设置, 最后制定路由Key
		channel.exchangeDeclare(exchangeName, "topic", true);
		channel.queueDeclare(queueName, true, false, false, null);
		channel.queueBind(queueName, exchangeName, routingKey);
		
		//4 创建消费者 
		QueueingConsumer queueingConsumer = new QueueingConsumer(channel);
		channel.basicConsume(queueName, true, queueingConsumer);
		
		while(true){
			Delivery delivery = queueingConsumer.nextDelivery();
			String msg = new String(delivery.getBody());
			
			System.err.println("消费端: " + msg);
		}
		
		
	}
}



复制代码

Tools:


/**
 * 
* @ClassName: ConnectionUtils 
* @Description: 连接工具类
* @author Coder编程
* @date 2019年6月21日 上午22:28:22 
*
 */
public class ConnectionUtils {
    public static Connection getConnection() throws IOException, TimeoutException {
        //定义连接工厂
        ConnectionFactory factory = new ConnectionFactory();
        //设置服务地址
        factory.setHost("127.0.0.1");
        //端口
        factory.setPort(5672);//amqp协议 端口 类似与mysql的3306
        //设置账号信息，用户名、密码、vhost
        factory.setVirtualHost("/vhost_cp");
        factory.setUsername("user_cp");
        factory.setPassword("123456");
        // 通过工程获取连接
        Connection connection = factory.newConnection();
        return connection;
    }
}


复制代码

First start consumer end = "restart the production side

3.3 View control station:

3.4 Print Results:

It can be observed that the consumer side of the first message is received, after the production side again receive callback information. If the disk is full, RabbitMQ abnormal, queue capacity limit is reached are likely to receive no ack

If no ack ack and none received the message, which is said before. RabbitMQ network glitch occurs, it may be employed the above-mentioned message compensation .

4. Return the message mechanism

Return Listener for processing some of the non-route messages!
Our news producer, and by specifying an Exchange Routingkey, to deliver messages to a particular queue to go, and then we listen queue consumers, consume processing operations!
However, in some cases, if we have time to send a message, the current exchange does not exist or is less than the specified route key route, this time if we do not need to listen up to this message, use the Return Listener!

There is a critical CIs on the basis of the API:

Mandatory: If true, the listener will receive the routes unreachable message, and then follow-up treatment, if false, then the broker end automatically delete the message!

4.1 Return messaging process

Producer production side to send a message to the MQ Broker end, but appear NotFind Exchange, Exchange messages sent, could not find the end Broker. Or found, but less than the specified routing key routing queue. So is the wrong message. This time, the production side should know that this message is sent, and will not be processed. Thus Return MQ Broker provides this mechanism to send these unreachable message to the production side, it is necessary at this time produce an end disposed to receive the Return The Listener unreachable message. Then in time logging, to process these messages.

4.2 code shows

Producer:


/**
 * 
* @ClassName: Producer 
* @Description: 生产者
* @author Coder编程
* @date 2019年7月30日 上午22:03:22 
*
 */
public class Producer {

	
	public static void main(String[] args) throws Exception {
		
		
		//1 创建ConnectionFactory
		Connection connection = ConnectionUtils.getConnection();
		
		Channel channel = connection.createChannel();
		
		String exchange = "test_return_exchange";
		String routingKey = "return.save";
		String routingKeyError = "abc.save";
		
		String msg = "Hello RabbitMQ Return Message";
		
		
		channel.addReturnListener(new ReturnListener() {
			@Override
			public void handleReturn(int replyCode, String replyText, String exchange,
					String routingKey, AMQP.BasicProperties properties, byte[] body) throws IOException {
				
				System.err.println("---------handle  return----------");
				//响应码
				System.err.println("replyCode: " + replyCode);
				//响应文本
				System.err.println("replyText: " + replyText);
				System.err.println("exchange: " + exchange);
				System.err.println("routingKey: " + routingKey);
				System.err.println("properties: " + properties);
				System.err.println("body: " + new String(body));
			}
		});
		
		//第三个参数mandatory=true,意味着路由不到的话mq也不会删除消息,false则会自动删除
		channel.basicPublish(exchange, routingKey, true, null, msg.getBytes());
		//修改routingkey，测试是否能够收到消息
		//channel.basicPublish(exchange, routingKeyError, true, null, msg.getBytes());
	}
}

复制代码

consumer:


/**
 * 
* @ClassName: Consumer 
* @Description: 消费者
* @author Coder编程
* @date 2019年7月30日 上午22:33:34 
*
 */
public class Consumer {

	
	public static void main(String[] args) throws Exception {
		
		
		//1 创建ConnectionFactory
		Connection connection = ConnectionUtils.getConnection();
		
		Channel channel = connection.createChannel();
		
		String exchangeName = "test_return_exchange";
		String routingKey = "return.#";
		String queueName = "test_return_queue";
		
		channel.exchangeDeclare(exchangeName, "topic", true, false, null);
		channel.queueDeclare(queueName, true, false, false, null);
		channel.queueBind(queueName, exchangeName, routingKey);
		
		QueueingConsumer queueingConsumer = new QueueingConsumer(channel);
		
		channel.basicConsume(queueName, true, queueingConsumer);
		
		while(true){
			Delivery delivery = queueingConsumer.nextDelivery();
			String msg = new String(delivery.getBody());
			System.err.println("消费者: " + msg);
		}
	}
}


复制代码

ConnectionUtils tool in the above code.

Start consumer side, and view the control station.

4.3 View control station

4.4 Print View results

Consumption release side code: channel.basicPublish (exchange, routingKey, true, null, msg.getBytes ()); consumer side printing results:

Normal printing result can be seen, this time to change the code to: channel.basicPublish (exchange, routingKeyError, true, null, msg.getBytes ());

We can see the production receives an unreachable messages.

The end of the sentence

Welcome attention to personal micro-channel public number: Coder program for the latest original technical articles and free learning materials, a lot more boutique mind maps, interview data, PMP preparation materials waiting for you to lead, allowing you to learn technical knowledge anytime, anywhere! Create a qq group: 315 211 365, we welcome into the group exchange study together. Thank you! Can also be introduced to the side of a friend in need.

Articles included to Github: github.com/CoderMerlin... Gitee: gitee.com/573059382/c... welcome attention and star ~

Reference article:

"RabbitMQ messaging middleware succinctly"