1. Problem background
For the monitoring of the message queue, we generally use Java to write an independent program and run it on the Linux server. After the program is started, the message is received through the message queue client and placed in a thread pool for asynchronous processing and concurrent fast processing.
So the question is, when we need to restart the task after modifying the program, how to ensure that the message is not lost?
Normally, after the subscriber program is closed, messages will accumulate in the sender queue, waiting for the next subscription consumption of the subscriber, so unreceived messages will not be lost. The only messages that may be lost are those that have been removed from the queue at the moment of shutdown but have not yet been processed.
Therefore, we need a smooth shutdown mechanism to ensure that messages can be processed normally when restarting.
2. Problem analysis
The idea of smooth closing is as follows:
- When closing the program, first close the message subscription, this time the messages are all in the sender queue
- Close the local message processing thread pool (waiting for messages in the local thread pool to be processed)
- program exit
Closing the message subscription: Generally, the client of the message queue provides a method to close the connection. For details, you can check the api by yourself.
Close the thread pool: Java's ThreadPoolExecutorthread pool provides shutdown()and shutdownNow()two methods, the difference is that the former will wait for the messages in the thread pool to be processed, and the latter will directly stop the execution of the thread and return to the list collection. Because we need to use the shutdown()method to close, and through isTerminated()the method to determine whether the thread pool has been closed.
So the question comes again, how do we notify the program that a shutdown operation needs to be performed?
In Linux, we can use to kill -9 pidclose the process, in addition to -9, we can kill -lcheck other semaphores of the kill command, such as using 12) SIGUSR2 semaphore
We can register the corresponding semaphore when the Java program starts, monitor the semaphore, and perform related business operations when the corresponding kill operation is received.
The pseudo code is as follows
//注册linux kill信号量 kill -12
Signal sig = new Signal("USR2");
Signal.handle(sig, new SignalHandler() {
@Override
publicvoidhandle(Signal signal){
//关闭订阅者
//关闭线程池
//退出
}
});
The following is a demo to simulate the relevant logic operations
First simulate a producer that produces 5 messages per second
Then simulate a subscriber, and hand it over to the thread pool for processing after receiving the message. The thread pool is fixed with 4 threads, and each message processing time is 1 second, so that the thread pool will have a backlog of 1 message per second.
package com.lujianing.demo;
import sun.misc.Signal;
import sun.misc.SignalHandler;
import java.util.concurrent.*;
/**
* @author [email protected]
* @Description:
* @date 2016/11/14
*/
public classMsgClient{
//模拟消息队列订阅者 同时4个线程处理
private static final ThreadPoolExecutor THREAD_POOL = (ThreadPoolExecutor) Executors.newFixedThreadPool(4);
//模拟消息队列生产者
private static final ScheduledExecutorService SCHEDULED_EXECUTOR_SERVICE = Executors.newSingleThreadScheduledExecutor();
//用于判断是否关闭订阅
private static volatile boolean isClose = false;
publicstaticvoidmain(String[] args)throws InterruptedException {
BlockingQueue <String> queue = new ArrayBlockingQueue<String>(100);
producer(queue);
consumer(queue);
}
//模拟消息队列生产者
privatestaticvoidproducer(final BlockingQueue queue){
//每200毫秒向队列中放入一个消息
SCHEDULED_EXECUTOR_SERVICE.scheduleAtFixedRate(new Runnable() {
publicvoidrun(){
queue.offer("");
}
}, 0L, 200L, TimeUnit.MILLISECONDS);
}
//模拟消息队列消费者 生产者每秒生产5个 消费者4个线程消费1个1秒 每秒积压1个
privatestaticvoidconsumer(final BlockingQueue queue)throws InterruptedException {
while (!isClose){
getPoolBacklogSize();
//从队列中拿到消息
final String msg = (String)queue.take();
//放入线程池处理
if(!THREAD_POOL.isShutdown()) {
THREAD_POOL.execute(new Runnable() {
publicvoidrun(){
try {
//System.out.println(msg);
TimeUnit.MILLISECONDS.sleep(1000L);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
});
}
}
}
//查看线程池堆积消息个数
privatestaticlonggetPoolBacklogSize(){
long backlog = THREAD_POOL.getTaskCount()- THREAD_POOL.getCompletedTaskCount();
System.out.println(String.format("[%s]THREAD_POOL backlog:%s",System.currentTimeMillis(),backlog));
return backlog;
}
static {
String osName = System.getProperty("os.name").toLowerCase();
if(osName != null && osName.indexOf("window") == -1) {
//注册linux kill信号量 kill -12
Signal sig = new Signal("USR2");
Signal.handle(sig, new SignalHandler() {
@Override
publicvoidhandle(Signal signal){
System.out.println("收到kill消息，执行关闭操作");
//关闭订阅消费
isClose = true;
//关闭线程池，等待线程池积压消息处理
THREAD_POOL.shutdown();
//判断线程池是否关闭
while (!THREAD_POOL.isTerminated()) {
try {
//每200毫秒 判断线程池积压数量
getPoolBacklogSize();
TimeUnit.MILLISECONDS.sleep(200L);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
System.out.println("订阅者关闭，线程池处理完毕");
System.exit(0);
}
});
}
}
}
When we run on the service, we can see the relevant output information through the console, and the number of backlog messages of the thread pool is output in the demo.
java -cp /home/work/lujianing/msg-queue-client/* com.lujianing.demo.MsgClient
Open another terminal and view the process ID through the ps command, or start the Java process through nohup to get the process ID
ps -fe|grep MsgClient
When we execute kill -12 pid, we can see the shutdown business logic
3. Summary of the problem
In the actual business of the department, the message volume of the message queue is still quite large. In some business peaks, there are hundreds of messages per second. Therefore, the speed of message processing must be guaranteed to avoid message backlog. Subscription node pressure.
In some business scenarios, the message integrity requirements are not so high, so there is no need to consider a little loss during restart. On the contrary, it requires careful thinking and design.
ps: I will resume the habit of blogging in the future, and strive to update it every week. If you are interested, please pay attention
2016-11-17 Supplement
ThreadPoolExecutormethod, which returns the getQueue().size()number of messages backlogged in the thread pool queue
getTaskCount() - getCompletedTaskCount(), returns the number of backlogged and in-process messages for the thread pool queue
2016-11-19 Supplement
Thanks @Nalan Qingfeng for the reminder
Runtime.getRuntime().addShutdownHook()In Java, a callback can be triggered when the JVM exits by calling a method
kill -15 pid Can trigger the call to the corresponding hook method
Today I look at kill -15the but I don't know if the two are related.
2016-11-20 Supplement
kill -1 pid and kill -2 pid both trigger hook methods
SIGTERM, SIGINT, SIGHUP three signals will trigger shutdownhook
2016-11-27 Supplement
In the company's RPC service framework, the same smooth shutdown strategy is followed.
Set the shutdown state of the rpc framework through kill -12, and close the related connection through kill -15