Java Notes - Practice and Thinking of RabbitMQ

foreword

This essay will summarize some of my understanding of the message queue RabbitMQ, and by the way, talk about its specific application in high concurrency and spike systems.

1. Preliminary example

After thinking about it, it is better to throw a simple example first, and then expand it according to its specific application scenario. I think it is more coherent.

RabbitConfig：

@Configuration
public class RabbitConfig {

    @Bean
    public Queue callQueue() {
        return new Queue(MQConstant.CALL);
    }
}

Client：

@Component
public class Client {

    @Autowired
    private RabbitTemplate rabbitTemplate;

    public void sendCall(String content) {
        for (int i = 0; i < 10000; i++) {
            String message = i + "-" + content;
            System.out.println(String.format("Sender: %s", message));
            rabbitTemplate.convertAndSend(MQConstant.CALL, message);
        }
    }
}

Server：

@Component
public class Server {

    @RabbitHandler
    @RabbitListener(queues = MQConstant.CALL)
    public void callProcess(String message) throws InterruptedException {
        Thread.sleep(1000);
        System.out.println(String.format("Receiver: reply(\"%s\") Yes, I just saw your message!", message));
    }

}

Result：

Sender: Hello, are you there!
Receiver: reply("Hello, are you there!") Yes, I just saw your message!

The above example will create a queue CALL in rabbitmq, where messages are waiting to be consumed:

For simple extensions on this basis, I will no longer write cases. For example, after the domain module completes its core business rules, it may need to update the cache, write an email, record a complex log, make a statistical report, etc. These do not need to be timely. Feedback or time-consuming auxiliary services can be distributed through asynchronous queues to improve the response speed of the core business. At the same time, this processing can make the domain boundary clearer, and the maintainability of the code and the ability to continuously expand will also be improved.

2. Peak clipping

The application scenario I mentioned in the previous example is decoupling and notification, and then extended. Because of its good buffering properties, another very suitable application scenario is peak shaving. For sudden extremely high concurrent requests, we can quickly queue them up and reply with a friendly prompt to the user, and then the server can process them slowly within the range that they can bear, thus preventing bursts of CPU and memory" Explosion".

After the transformation, it is of course relatively cool for the sender. He just adds the request to the message queue, and the processing pressure is attributed to the consumer. Then think about it, are there any side effects of doing this? If this request happens to be blocked by the thread, then it has to be added to the queue and slowly queued for processing. That is not the end, and the user needs to be able to get feedback? So in response to this, I think the method on the consumer side should be changed to asynchronous calls (ie multi-threading) to improve throughput, and the way it is written in Spring Boot is also very simple:

@Component
public class Server {

    @Async
    @RabbitHandler
    @RabbitListener(queues = MQConstant.CALL)
    public void callProcess(String message) throws InterruptedException {
        Thread.sleep(100);
        System.out.println(String.format("Receiver: reply(\"%s\") Yes, I just saw your message!", message));
    }

}

Referring to the method in Example 1, I posted 10,000 messages to the queue, and the call on the consumer side blocks for one second each time. That's interesting. When will it be processed? However, if hundreds of threads are opened to process at the same time, tens of seconds is enough. Of course, how much is appropriate should be considered according to the specific business scenario and server configuration. Also, don't forget to configure the thread pool:

@Configuration
public class AsyncConfig {

    @Bean
    public Executor asyncExecutor(){
        ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
        executor.setCorePoolSize(10);
        executor.setMaxPoolSize(500);
        executor.setQueueCapacity(10);

        executor.setThreadNamePrefix("MyExecutor-");

        executor.setRejectedExecutionHandler(new ThreadPoolExecutor.CallerRunsPolicy());
        executor.initialize();
        return executor;
    }
}

3. Exchange

RabbitMQ may provide services for N applications at the same time. If you and your blue-eyed confidant suddenly have a heart-to-heart and use the same routingKey for different businesses, it will be exciting to think about it. Therefore, if there are more queues, it is natural to conduct group management, define the rules of Exchange, and then you can play alone.

MQConstant：

public class MQConstant {

    public static final String EXCHANGE = "YOUCLK-MESSAGE-EXCHANGE";

    public static final String CALL = MQConstant.EXCHANGE + ".CALL";

    public static final String ALL = MQConstant.EXCHANGE + ".#";
}

RabbitConfig：

@Configuration
public class RabbitConfig {

    @Bean
    public Queue callQueue() {
        return new Queue(MQConstant.CALL);
    }

    @Bean
    TopicExchange exchange() {
        return new TopicExchange(MQConstant.EXCHANGE);
    }

    @Bean
    Binding bindingExchangeMessage(Queue queueMessage, TopicExchange exchange) {
        return BindingBuilder.bind(queueMessage).to(exchange).with(MQConstant.ALL);
    }
}

At this point, we check the queue CALL again, and we can see that Exchange has been bound:

Of course, the role of Exchange is much more than that. The above example is the Topic mode, in addition to the Direct, Headers and Fanout modes, which are written in the same way. Interested children can go to the " official document " for a more in-depth understanding.

4. Delay queue

I believe that the scene of the delayed task has been touched by all the friends, especially when the order is rushed, the order that has not been paid within the specified time will be recycled. The WeChat payment API also has a delayed re-confirmation message push after the payment is completed, and the implementation principles should be similar.

To use RabbitMQ to implement this function, you must first understand its two features, namely Time-To-Live Extensions and Dead Letter Exchanges, which can be roughly understood literally, one is the survival time, and the other is the dead letter. The whole process is also easy to understand. TTL is equivalent to a buffer queue. After waiting for its expiration, the message will be forwarded by DLX to the actual consumption queue, thus realizing his delay process.

MQConstant：

public class MQConstant {

    public static final String PER_DELAY_EXCHANGE = "PER_DELAY_EXCHANGE";

    public static final String DELAY_EXCHANGE = "DELAY_EXCHANGE";

    public static final String DELAY_CALL_TTL = "DELAY_CALL_TTL";

    public static final String CALL = "CALL";

}

ExpirationMessagePostProcessor：

public class ExpirationMessagePostProcessor implements MessagePostProcessor {
    private final Long ttl;

    public ExpirationMessagePostProcessor(Long ttl) {
        this.ttl = ttl;
    }

    @Override
    public Message postProcessMessage(Message message) throws AmqpException {
        message.getMessageProperties()
                .setExpiration(ttl.toString());
        return message;
    }
}

Client：

@Component
public class Client {

    @Autowired
    private RabbitTemplate rabbitTemplate;

    public void sendCall(String content) {
        for (int i = 1; i <= 3; i++) {
            long expiration = i * 5000;
            String message = i + "-" + content;
            System.out.println(String.format("Sender: %s", message));
            rabbitTemplate.convertAndSend(MQConstant.DELAY_CALL_TTL, (Object) message, new ExpirationMessagePostProcessor(expiration));

        }
    }
}

Server：

@Component
public class Server {

    @Async
    @RabbitHandler
    @RabbitListener(queues = MQConstant.CALL)
    public void callProcess(String message) throws InterruptedException {
        String date = (new SimpleDateFormat("HH:mm:ss")).format(new Date());
        System.out.println(String.format("Receiver: reply(\"%s\") Yes, I just saw your message!- %s", message, date));
    }

}

Result：

Sender: 1-Hello, are you there!
Sender: 2-Hello, are you there!
Sender: 3-Hello, are you there!
Receiver: reply("1-Hello, are you there!") Yes, I just saw your message!- 23:04:12
Receiver: reply("2-Hello, are you there!") Yes, I just saw your message!- 23:04:17
Receiver: reply("3-Hello, are you there!") Yes, I just saw your message!- 23:04:22

The results are clear at a glance. The queues are delayed by 5 seconds, 10 seconds, and 15 seconds. Of course, the above is just my simple example. Children's shoes can read the official documents (" ttl " && " dlx ") for further study.

I have a WeChat public account, and I often share some dry goods related to Java technology; if you like my sharing, you can use WeChat to search for "Java Head" or "javatuanzhang" to follow.