RabbitMQ from entry to the master (two)

1. How successful message delivery guarantee one hundred percent?

What is the reliability of the delivery end of the production?

• Ensure the success of the message sent
• MQ nodes successfully receive protection
• MQ sender receives node (Broker) acknowledgment
• Improve the compensation mechanism for message

If you want to guarantee message delivery hundred percent success, only the first three steps do not necessarily guaranteed. Some time or that some extreme cases, such as the production side when delivering the message may have failed, or that the production side delivered the news, MQ also received, MQ when returning the acknowledgment, due to network glitches led to the production does not receive answer, do not know at this time the message delivery success or failure, so for those cases we need to do some compensation mechanism.

 

1.1 Option One: Message for falling, the message status marking

 

1. Storage of data, such as our order form to send a message, first of all the business data warehousing is, order information, and then generates a message, the message can also be put in storage, this message should contain the message state attribute Create_Date (Created ), and set the initial signs such as 0, it indicates that the message is successfully created, being sent in
2. First, to ensure the success of the first step messages are stored, without any exceptions, then the production side and then send a message. If you fail to quick failure mechanism
3. The result MQ messages received response (confirm)to the production end

4. Producing a terminal Confirm Listener, asynchronous to monitor Brokerthe response sent back, to determine whether the message is successfully delivered, if successful, the message to query the database, and updates a status message, a message indicating successful delivery
    
   assumed OK the second step, the first when the three-echo response, the network suddenly appeared a flash, resulting in the production side of the Listener will never receive confirm this news answer, and that is the status of this news has been the 0

5. At this point we need to set a rule, such as message sets a threshold timeout in storage time, 5 minutes or 0 if the status of the message it would need to be extracted. We use here is distributed regular tasks, the timing to fetch the DB was created more than five minutes and status messages to the message from zero.
6. To crawl out of the news re-delivery (Retry Send), that is, he began to continue to go down from the second step

7. Of course, some messages might be due to some practical problems can not be routed to the Broker, such as routingKey not set up, the corresponding queue is deleted by mistake, then this message even if retry several times and still can not be delivered successfully, it needs to be done on the number of retries restrictions, such as restrictions three times, if the delivery number is greater than three times, then the message status will be updated to 2, indicating that the final delivery failure messages.

 

In view of this situation how to do compensation for it, you can have a compensation system to query the ultimate failure of the message, and then gives the reason for the failure, of course, these may need to operate manually.

The first delivery reliability, suitability in a highly concurrent scenarios?

For the first scenario, we need to do the database persistence operations twice, apparently database performance bottleneck in high concurrency scenarios. In fact, only need to be in our core business link in storage on it, the message is no need to put in storage, we can do delayed delivery of the message, do the second confirmation callback inspection.

Of course, this scheme does not necessarily guarantee hundred percent successful delivery, but basically you can guarantee about 99.9% of the message is OK, some particularly extreme cases can only be compensated to do it manually, or use scheduled tasks can be done .

 

1.2 Option II: delay delivery of the message, do the second acknowledgment, checking the callback

 

Upstream ServiceUpstream service is the production side, Downstream servicethe downstream end of the service is consumption, Callback serviceis a callback service.

 

1. First storage service message, and then sends out the end of message production
2. After sending the message, followed by a message transmitting end production again (Second Send Delay Check), i.e. delayed message delivery inspection, there need to set a delay time, such as for delivery after 5 minutes.
3. The consumer side to listen on the specified queue, the message is received for processing.
4. After the process is completed, sending a confirmmessage, the response is sent back, but this is not a normal response to the ACK, but re-generates a message delivered to the MQ.
5. The above Callback serviceis a separate service, in fact, it plays the role of DB storing messages first scenario, which is to monitor downstream services sent by MQ confirmmessage, if the Callback servicereceived confirmmessage, then the message do persistent storage, the message is about persisted to the DB.
6. After 5 minutes the MQ message to the transmission delay, and Callback serviceafter a delay or listen to the message queue corresponding to the received message Check message DB to check if there are, if there is no need to do any processing, if there is no consumption or failure , then Callback serviceyou need to initiate communications to the upstream RPC service, tell it to delay check I did not find this message, you need to re-send, receive information after the production side will re-query service message and then send out the message.

The goal is less to do with the storage of a DB in high concurrency scenarios, most concerned about is not the message delivered 100 percent success, but we must ensure the performance, guaranteed hard to resist such a large amount of concurrency. It is possible to save as much as possible to save the database operation may be asynchronous compensated.

 

In fact, there is no main flow of the Callback service, it belongs to a compensation of service, the entire core link is the production end storage business news, send a message to MQ, the queue monitor consumer side, consumer news. Other steps are a compensation mechanism.

The second option is the Internet giant is more classic and mainstream solutions. But if not so high performance requirements, the first program to be simpler

 

2. Idempotence

2.1 What sex is power and so on?

Is simply the result of a request for a user or multiple requests of the same operation initiated is the same.

We can learn from optimistic locking mechanism of the database to give an example:

• First, add a version field for the table version

• Before the update operation it will go to the database query this version

• Then perform the update statement to version as a condition, such as:

  UPDATE T_REPS SET COUNT = COUNT -1，VERSION = VERSION + 1 WHERE VERSION = 1

• There are other people to update the data in this form if you execute the update, then this condition will not come into force, it will not perform the operation, and to protect the idempotency this optimistic locking mechanism.

 

2.2 Message end idempotency guarantee

Repeat consumer issues:

When finished consumer news consumption, while the production side returns ack due to network outages, leading to the production side did not receive a confirmation message, the message will re-send the article and consumer spending, but in fact the consumer has successfully consumed news article, which is repeated consumption problem.

 

2.2.1 The only ID + fingerprint code mechanism

Unique ID: business unique primary key table, such as commodity ID

Fingerprint code: In order to distinguish each normal operation code, fingerprint code generated at each operation; + service ID may be a time stamp or a flag (traffic depending on scenes)

 

• ID + fingerprint code unique mechanism, the primary key to re-use database
• SELECT COUNT (1) FROM T_ORDER WHERE ID = unique fingerprint code ID and IS_CONSUM =
• Benefits: simple
• Disadvantages: There are performance bottlenecks database write under high concurrency
• Solution: The library is divided routing algorithm based on sub-table ID

The whole idea is, first of all we need to generate a globally unique ID based on the message, and then also need to add a fingerprint code. The fingerprint code it does not have to be generated by the system, but some external or internal rules of business rules to splicing, its purpose is to protect this operation is absolutely unique.

The spliced ​​ID + fingerprint code as a database primary key value, the de-duplication can be performed. That is the message it before consumption, go to the message database query fingerprint identification code is present, it does not perform insert operation, if there has been expended on behalf of, they do not have control of.

 

2.2.2 using the Redis to achieve atomicity

Here only mentioned with atomic Redis to solve the problem MQ idempotency repeat consumption

Note: MQ idempotency problem lies in the production end of the ACK is not received properly, it may be the network jitter, resulting in network outages

 

My plan:

MQ when the consumer side in the consumer ID start to put in the Redis BitMap, MQ production data for each end of production, from a position corresponding to the Redis BitMap if not taken ID, the production messaging, message transmission or not.

But someone might say, if the consumer side, the production side Redis command fails how to do, although the possibility has emerged repeat consumption appeared abnormal Redis command execution is extremely low, but in case it?

OK, we can in the Redis command fails, the message off the library, daily timer, on this very special message for processing.

 

3. Confirm mechanism

3.1 How to understand?

• Confirmation message refers to the producer after the delivery of the message, if Broker receives a message, it will give us a reply producer
• The producer receives the response, to determine whether the message is transmitted normally to the Broker, this approach is the reliability of message delivery

The core protection

 

Confirm flowchart mechanism

Production sends a message to the Broker, then the Broker receiving the message, the echo response, the production side a Confirm Listener, to snoop response, of course, this operation is asynchronous, the production side sends the message out can not control, so that the internal monitor Broker is to listen to our response.

 

3.2 how to achieve?

• The first step, open the confirmation mode on the channel:channel.confirmSelect()
• 2. Add listening on Channel: addConfirmListener, returns the success and failure results of listening, resend the message according to a specific result, the subsequent processing, or the like logging!

public class Producer {
    public static void main(String[] args) throws Exception {
        
        //创建ConnectionFactory
        ConnectionFactory connectionFactory = new ConnectionFactory();
        connectionFactory.setHost("192.168.244.11");
        connectionFactory.setPort(5672);
        connectionFactory.setVirtualHost("/");
        connectionFactory.setHandshakeTimeout(20000);

        
        //获取Connection
        Connection connection = connectionFactory.newConnection();
        
        //通过connection创建一个新的Channel
        Channel channel = connection.createChannel();
        
        //指定我们的消息投递模式
        channel.confirmSelect();
        
        String exchangeName = "test_confirm_exchange";
        String routingkey = "confirm.save";
        
        //发送一条信息
        String msg = "Hello RabbitMQ Send confirm message!";
        channel.basicPublish(exchangeName, routingkey, null, msg.getBytes());
        
        //添加一个确认监听
        channel.addConfirmListener(new ConfirmListener() {
            
            @Override
            public void handleNack(long deliveryTag, boolean multiple)
                    throws IOException {
                System.out.println("-------no ack!---------");
            }
            
            @Override
            public void handleAck(long deliveryTag, boolean multiple)
                    throws IOException {
                System.out.println("--------ack!----------");
            }
        });
    }
}

 

public class Consumer {
    public static void main(String[] args) throws Exception{
        //创建ConnectionFactory
        ConnectionFactory connectionFactory = new ConnectionFactory();
        connectionFactory.setHost("192.168.244.11");
        connectionFactory.setPort(5672);
        connectionFactory.setVirtualHost("/");
        connectionFactory.setHandshakeTimeout(20000);

        
        //获取Connection
        Connection connection = connectionFactory.newConnection();
        
        //通过connection创建一个新的Channel
        Channel channel = connection.createChannel();
        
        String exchangeName = "test_confirm_exchange";
        String routingkey = "confirm.#";
        String queueName = "test_confirm_queue"; 
        
        //声明交换机和队列 然后进行绑定和 设置 最后制定路由key
        channel.exchangeDeclare(exchangeName, "topic",true);
        channel.queueDeclare(queueName, true, false, false, null);
        
        channel.queueBind(queueName, exchangeName, routingkey);
        
        //创建消费者
        QueueingConsumer queueingConsumer = new QueueingConsumer(channel);
        channel.basicConsume(queueName, true,queueingConsumer); 
        
        while(true){
            Delivery delivery = queueingConsumer.nextDelivery();
            String msg = new String(delivery.getBody());
            System.out.println("消费端：" + msg);
        }
    }
}

 

Operating instructions

First start the consumer side, access control units: http: // ip: 15672, inspection and Exchange Queue is set OK, and then start the production side, the message is the consumer end consumer, the production side but also to monitor the success of the ACK response.

 

4. Return Mechanism

4.1 How to understand?

• Return Listener Processing a message of some non-routable!
• Our news producer, and by specifying an Exchange Routingkey, to deliver messages to a particular queue to go, and then we listen queue consumers, consume processing operations!
• However, in some cases, if we have time to send a message, the current exchange does not exist or is less than the specified route key route, this time if we do not need to listen up to this message, it is necessary to use Return Listener!
  

 

4.2 How to achieve?

1. Add return listener: addReturnListenerthe production end to listen for these unreachable messages, to do some post-processing, for example, record the message log, or in a timely manner to track record, it is possible to re-set look like a
2. 发送消息时，设置Mandatory：如果为true，则监听器会接收到路由不可达的消息，然后进行后续处理，如果为false，那么broker端自动删除该消息！

 

public class ReturnProducer {
     public static void main(String[] args) throws Exception {
            //1 创建ConnectionFactory
            ConnectionFactory connectionFactory = new ConnectionFactory();
            connectionFactory.setHost("192.168.244.11");
            connectionFactory.setPort(5672);
            connectionFactory.setVirtualHost("/");
            connectionFactory.setHandshakeTimeout(20000);
            //2 获取Connection
            Connection connection = connectionFactory.newConnection();
            //3 通过Connection创建一个新的Channel
            Channel channel = connection.createChannel();
            
            String exchange = "test_return_exchange";
            //String routingKey = "return.save";
            String routingKeyError = "abc.save";
            
            String msg = "Hello RabbitMQ Return Message";
            //添加return监听
            channel.addReturnListener(new ReturnListener() {
                @Override
                public void handleReturn(int replyCode, String replyText, String exchange,
                        String routingKey, BasicProperties properties, byte[] body) throws IOException {
                    //replyCode：响应码    replyText：响应信息
                    System.err.println("---------handle  return----------");
                    System.err.println("replyCode: " + replyCode);
                    System.err.println("replyText: " + replyText);
                    System.err.println("exchange: " + exchange);
                    System.err.println("routingKey: " + routingKey);
                    //System.err.println("properties: " + properties);
                    System.err.println("body: " + new String(body));
                }

                
            });
            //5 发送一条消息，第三个参数mandatory：必须设置为true
            channel.basicPublish(exchange, routingKeyError, true, null, msg.getBytes());
        }
}

 

public class ReturnConsumer {
    
    public static void main(String[] args) throws Exception {
        //1 创建ConnectionFactory
        ConnectionFactory connectionFactory = new ConnectionFactory();
        connectionFactory.setHost("192.168.244.11");
        connectionFactory.setPort(5672);
        connectionFactory.setVirtualHost("/");
        connectionFactory.setHandshakeTimeout(20000);
        //2 获取Connection
        Connection connection = connectionFactory.newConnection();
        //3 通过Connection创建一个新的Channel
        Channel channel = connection.createChannel();
        
        String exchangeName = "test_return_exchange";
        String routingKey = "return.#";
        String queueName = "test_return_queue";
        //4 声明交换机和队列，然后进行绑定设置路由Key
        channel.exchangeDeclare(exchangeName, "topic", true, false, null);
        channel.queueDeclare(queueName, true, false, false, null);
        channel.queueBind(queueName, exchangeName, routingKey);
        
        //5 创建消费者 
        QueueingConsumer queueingConsumer = new QueueingConsumer(channel);
        channel.basicConsume(queueName, true, queueingConsumer);
        
        while(true){
            Delivery delivery = queueingConsumer.nextDelivery();
            String msg = new String(delivery.getBody());
            System.err.println("消费者: " + msg);
        }
    }
}

 

运行说明

先启动消费端，访问管控台：http://ip:15672，检查Exchange和Queue是否设置OK，然后启动生产端。
由于生产端设置的是一个错误的路由key，所以消费端没有任何打印，而生产端打印了如下内容

如果我们将 Mandatory 属性设置为false，对于不可达的消息会被Broker直接删除，那么生产端就不会进行任何打印了。如果我们的路由key设置为正确的，那么消费端能够正确消费，生产端也不会进行任何打印。