1 Introduction
emm, I have encountered another problem. There is a window period for the application of the existing business system to go online , which cannot meet the iterative launch of normal tasks. Going online during the non-window period will easily lead to interruption of database, mq, jsf and other threads, which will lead to the need to manually repair the order. Therefore, it is optimized by adding the graceful shutdown function, so that after it chooses graceful shutdown before going online, the influx of new traffic will be cut off first, and a certain period of time will be reserved to process existing connections, and finally it will be completely offline, which can effectively expand the reservation window for online Time and reduce thread interruptions during go-live, thereby reducing manual order repairs. But what is graceful shutdown? Why does the existing system technology not have a native graceful shutdown mechanism? The research papers are organized as follows.
2. What is graceful shutdown?
•
Graceful shutdown refers to notifying the application process to release the occupied resources when the application is closed.
•
, shutdown (do not accept new tasks and wait for processing) or shutdownNow (call Thread.interrupt to interrupt).
线程池
•
socket connection, such as: netty, . ( Need to focus on )
jmq、fmq
•
Inform registry to go offline quickly, eg . ( Need to focus on )
jsf
•
Clean up temporary files.
•
Various in-heap and off-heap memory releases.
In short, the forced termination of the process will cause data loss or the terminal cannot be restored to a normal state, which may lead to data inconsistency in a distributed environment.
3. The culprit that leads to elegant shutdown is not elegant - kill command
•
kill command
◦
kill -15 : The default value of the kill command is -15. The knowledge sends a signal to notify the process to terminate. What to do by the process, that is, the process does not necessarily terminate . Generally, kill -15 is not used directly, and the process may not be terminated.
SIGTERM自行决定
◦
kill -9 : Forcefully terminate the process, the process will be terminated immediately. kill -9 is too violent , and transaction execution and business processing are often interrupted, resulting in the existence of residual files in the database and the system. If you want to use kill -9, try to use kill -15 first to give the process a chance to deal with the aftermath.
脏数据该命令可以模拟一次系统宕机，系统断电等极端情况。
◦
kill -2 : Similar to Ctrl + C to exit, it will save relevant data first and then terminate the process. kill -2 Immediately -> -> , but the relevant data will be saved before the process is terminated, and transaction execution and business processing will still be interrupted, and graceful shutdown cannot be achieved.
终止正在执行的代码保存数据终止进程
4. Extended question: How does jvm accept and process linux semaphores?
•
The customization is loaded when the jvm starts , and the corresponding handle is triggered when the jvm is closed .
SingalHandler
public interface SignalHandler {
SignalHandler SIG_DFL = new NativeSignalHandler(0L);
SignalHandler SIG_IGN = new NativeSignalHandler(1L);
void handle(Signal var1);
}
class Terminator {
private static SignalHandler handler = null;
Terminator() {
}
//jvm设置SignalHandler，在System.initializeSystemClass中触发
static void setup() {
if (handler == null) {
SignalHandler var0 = new SignalHandler() {
public void handle(Signal var1) {
Shutdown.exit(var1.getNumber() + 128);//调用Shutdown.exit
}
};
handler = var0;
try {
Signal.handle(new Signal("INT"), var0);//中断时
} catch (IllegalArgumentException var3) {
}
try {
Signal.handle(new Signal("TERM"), var0);//终止时
} catch (IllegalArgumentException var2) {
}
}
}
}
•
Runtime.addShutdownHook . Before you understand , look at it first ; it is to add a shutdown hook to the jvm, which is called when the jvm is running.
Shutdown.exitRuntime.getRuntime().addShutdownHook(shutdownHook)关闭
public class Runtime {
public void addShutdownHook(Thread hook) {
SecurityManager sm = System.getSecurityManager();
if (sm != null) {
sm.checkPermission(new RuntimePermission("shutdownHooks"));
}
ApplicationShutdownHooks.add(hook);
}
}
class ApplicationShutdownHooks {
/* The set of registered hooks */
private static IdentityHashMap<Thread, Thread> hooks;
static synchronized void add(Thread hook) {
if(hooks == null)
throw new IllegalStateException("Shutdown in progress");
if (hook.isAlive())
throw new IllegalArgumentException("Hook already running");
if (hooks.containsKey(hook))
throw new IllegalArgumentException("Hook previously registered");
hooks.put(hook, hook);
}
}
//它含数据结构和逻辑管理虚拟机关闭序列
class Shutdown {
/* Shutdown 系列状态*/
private static final int RUNNING = 0;
private static final int HOOKS = 1;
private static final int FINALIZERS = 2;
private static int state = RUNNING;
/* 是否应该运行所以finalizers来exit? */
private static boolean runFinalizersOnExit = false;
// 系统关闭钩子注册一个预定义的插槽.
// 关闭钩子的列表如下:
// (0) Console restore hook
// (1) Application hooks
// (2) DeleteOnExit hook
private static final int MAX_SYSTEM_HOOKS = 10;
private static final Runnable[] hooks = new Runnable[MAX_SYSTEM_HOOKS];
// 当前运行关闭钩子的钩子的索引
private static int currentRunningHook = 0;
/* 前面的静态字段由这个锁保护 */
private static class Lock { };
private static Object lock = new Lock();
/* 为native halt方法提供锁对象 */
private static Object haltLock = new Lock();
static void add(int slot, boolean registerShutdownInProgress, Runnable hook) {
synchronized (lock) {
if (hooks[slot] != null)
throw new InternalError("Shutdown hook at slot " + slot + " already registered");
if (!registerShutdownInProgress) {//执行shutdown过程中不添加hook
if (state > RUNNING)//如果已经在执行shutdown操作不能添加hook
throw new IllegalStateException("Shutdown in progress");
} else {//如果hooks已经执行完毕不能再添加hook。如果正在执行hooks时，添加的槽点小于当前执行的槽点位置也不能添加
if (state > HOOKS || (state == HOOKS && slot <= currentRunningHook))
throw new IllegalStateException("Shutdown in progress");
}
hooks[slot] = hook;
}
}
/* 执行所有注册的hooks
*/
private static void runHooks() {
for (int i=0; i < MAX_SYSTEM_HOOKS; i++) {
try {
Runnable hook;
synchronized (lock) {
// acquire the lock to make sure the hook registered during
// shutdown is visible here.
currentRunningHook = i;
hook = hooks[i];
}
if (hook != null) hook.run();
} catch(Throwable t) {
if (t instanceof ThreadDeath) {
ThreadDeath td = (ThreadDeath)t;
throw td;
}
}
}
}
/* 关闭JVM的操作
*/
static void halt(int status) {
synchronized (haltLock) {
halt0(status);
}
}
//JNI方法
static native void halt0(int status);
// shutdown的执行顺序：runHooks > runFinalizersOnExit
private static void sequence() {
synchronized (lock) {
/* Guard against the possibility of a daemon thread invoking exit
* after DestroyJavaVM initiates the shutdown sequence
*/
if (state != HOOKS) return;
}
runHooks();
boolean rfoe;
synchronized (lock) {
state = FINALIZERS;
rfoe = runFinalizersOnExit;
}
if (rfoe) runAllFinalizers();
}
//Runtime.exit时执行，runHooks > runFinalizersOnExit > halt
static void exit(int status) {
boolean runMoreFinalizers = false;
synchronized (lock) {
if (status != 0) runFinalizersOnExit = false;
switch (state) {
case RUNNING: /* Initiate shutdown */
state = HOOKS;
break;
case HOOKS: /* Stall and halt */
break;
case FINALIZERS:
if (status != 0) {
/* Halt immediately on nonzero status */
halt(status);
} else {
/* Compatibility with old behavior:
* Run more finalizers and then halt
*/
runMoreFinalizers = runFinalizersOnExit;
}
break;
}
}
if (runMoreFinalizers) {
runAllFinalizers();
halt(status);
}
synchronized (Shutdown.class) {
/* Synchronize on the class object, causing any other thread
* that attempts to initiate shutdown to stall indefinitely
*/
sequence();
halt(status);
}
}
//shutdown操作，与exit不同的是不做halt操作(关闭JVM)
static void shutdown() {
synchronized (lock) {
switch (state) {
case RUNNING: /* Initiate shutdown */
state = HOOKS;
break;
case HOOKS: /* Stall and then return */
case FINALIZERS:
break;
}
}
synchronized (Shutdown.class) {
sequence();
}
}
}
5. How to achieve graceful shutdown in Spring?
•
Use Spring 3.2.12 to trigger some actions through events , mainly through . It can be seen that it is also based on the expansion.
springContexClosedEventLifecycleProcessor.onClosestopBeansspringjvm
public abstract class AbstractApplicationContext extends DefaultResourceLoader {
public void registerShutdownHook() {
if (this.shutdownHook == null) {
// No shutdown hook registered yet.
this.shutdownHook = new Thread() {
@Override
public void run() {
doClose();
}
};
Runtime.getRuntime().addShutdownHook(this.shutdownHook);
}
}
protected void doClose() {
boolean actuallyClose;
synchronized (this.activeMonitor) {
actuallyClose = this.active && !this.closed;
this.closed = true;
}
if (actuallyClose) {
if (logger.isInfoEnabled()) {
logger.info("Closing " + this);
}
LiveBeansView.unregisterApplicationContext(this);
try {
//发布应用内的关闭事件
publishEvent(new ContextClosedEvent(this));
}
catch (Throwable ex) {
logger.warn("Exception thrown from ApplicationListener handling ContextClosedEvent", ex);
}
// 停止所有的Lifecycle beans.
try {
getLifecycleProcessor().onClose();
}
catch (Throwable ex) {
logger.warn("Exception thrown from LifecycleProcessor on context close", ex);
}
// 销毁spring 的 BeanFactory可能会缓存单例的 Bean.
destroyBeans();
// 关闭当前应用上下文（BeanFactory）
closeBeanFactory();
// 执行子类的关闭逻辑
onClose();
synchronized (this.activeMonitor) {
this.active = false;
}
}
}
}
public interface LifecycleProcessor extends Lifecycle {
/**
* Notification of context refresh, e.g. for auto-starting components.
*/
void onRefresh();
/**
* Notification of context close phase, e.g. for auto-stopping components.
*/
void onClose();
}
6. How does SpringBoot achieve graceful shutdown?
•
Graceful shutdown is one of the features. After receiving the termination signal, it will no longer accept and process new requests, but will reserve a short buffer time before terminating the process, and the processing request has been completed. .
springboot注：优雅停机需要在tomcat的9.0.33及其之后的版本才支持
•
Some modules provide an interface for graceful shutdown. Execute the request . When the shutdown is successful, a prompt will be returned.
springbootspring-boot-starter-actuatorrestfulcurl -X POST http://127.0.0.1:8088/shutdown注：线上环境url需要设置权限，可配合spring-security使用火灾nginx限制内网访问。
#启用shutdown
endpoints.shutdown.enabled=true
#禁用密码验证
endpoints.shutdown.sensitive=false
#可统一指定所有endpoints的路径
management.context-path=/manage
#指定管理端口和IP
management.port=8088
management.address=127.0.0.1
#开启shutdown的安全验证（spring-security）
endpoints.shutdown.sensitive=true
#验证用户名
security.user.name=admin
#验证密码
security.user.password=secret
#角色
management.security.role=SUPERUSER
•
Implemented by calling .
springbootshutdownAbstractApplicationContext.close
@ConfigurationProperties(
prefix = "endpoints.shutdown"
)
public class ShutdownMvcEndpoint extends EndpointMvcAdapter {
public ShutdownMvcEndpoint(ShutdownEndpoint delegate) {
super(delegate);
}
//post请求
@PostMapping(
produces = {"application/vnd.spring-boot.actuator.v1+json", "application/json"}
)
@ResponseBody
public Object invoke() {
return !this.getDelegate().isEnabled() ? new ResponseEntity(Collections.singletonMap("message", "This endpoint is disabled"), HttpStatus.NOT_FOUND) : super.invoke();
}
}
@ConfigurationProperties(
prefix = "endpoints.shutdown"
)
public class ShutdownEndpoint extends AbstractEndpoint<Map<String, Object>> implements ApplicationContextAware {
private static final Map<String, Object> NO_CONTEXT_MESSAGE = Collections.unmodifiableMap(Collections.singletonMap("message", "No context to shutdown."));
private static final Map<String, Object> SHUTDOWN_MESSAGE = Collections.unmodifiableMap(Collections.singletonMap("message", "Shutting down, bye..."));
private ConfigurableApplicationContext context;
public ShutdownEndpoint() {
super("shutdown", true, false);
}
//执行关闭
public Map<String, Object> invoke() {
if (this.context == null) {
return NO_CONTEXT_MESSAGE;
} else {
boolean var6 = false;
Map var1;
class NamelessClass_1 implements Runnable {
NamelessClass_1() {
}
public void run() {
try {
Thread.sleep(500L);
} catch (InterruptedException var2) {
Thread.currentThread().interrupt();
}
//这个调用的就是AbstractApplicationContext.close
ShutdownEndpoint.this.context.close();
}
}
try {
var6 = true;
var1 = SHUTDOWN_MESSAGE;
var6 = false;
} finally {
if (var6) {
Thread thread = new Thread(new NamelessClass_1());
thread.setContextClassLoader(this.getClass().getClassLoader());
thread.start();
}
}
Thread thread = new Thread(new NamelessClass_1());
thread.setContextClassLoader(this.getClass().getClassLoader());
thread.start();
return var1;
}
}
}
7. The relationship between Tomcat and Spring for knowledge expansion?
By participating in the graceful downtime and refactoring of the cloud factory, it was found that there were problems with both, so the query explored the relationship between the two.TomcatSpring
•
He is the HTTP server and Servlet container , which is responsible for providing a running environment for servlets like Spring, among which: the functional boundary between the Http server and the Servlet container is: the HTTP server can be imagined as a reception, responsible for network communication and parsing requests , The Servlet container is the department responsible for processing business requests .
Tomcatjettey前台业务
•
Tomcat and Servlet, as a combination of Web server and Servlet container , can accept network http requests and parse them into request objects and response objects of Servlet specifications. For example, the HttpServletRequest object is provided by Tomcat, Servlet is a specification, Tomcat is a Servlet container that implements the specification, and SpringMVC is an application that processes Servlet requests. DispatcherServlet implements the Servlet interface, and Tomcat is responsible for loading and calling DispatcherServlet. At the same time, DispatcherServlet has its own container (SpringMVC) container, which is responsible for managing SpringMVC-related beans, such as Controler and ViewResolver. At the same time, there are other beans in Spring such as Service and DAO, which are managed by the global Spring IOC container. Therefore, Spring has two IOC containers.
•
If you just use spring (not including springmvc), then the tomcat container parses the xml file, instantiates the corresponding class through reflection, implements the class according to these servlet specifications, and triggers the corresponding code processing logic. At this time, tomcat is responsible for the analysis of the http message and servlet scheduling work.
•
If you use spring mvc, then tomcat just parses the http message, and then forwards it to dispatchsetvlet, and then springmvc executes the corresponding logic according to its configuration, the class corresponding to the instance, and then returns the result to dispatchservlet, and finally forwards it to tomcat , tomcat is responsible for constructing http message data.
8. Practical drills
•
( ) By adding a call to stop the consumption of the application when it is shut down , it prevents the occurrence of the thread pool during the online period .
mqjmq、fmqhookpausemq线程中断
/**
* @ClassName ShutDownHook
* @Description
* @Date 2022/10/28 17:47
**/
@Component
@Slf4j
public class ShutDownHook {
@Value("${shutdown.waitTime:10}")
private int waitTime;
@Resource
com.jdjr.fmq.client.consumer.MessageConsumer fmqMessageConsumer;
@Resource
com.jd.jmq.client.consumer.MessageConsumer jmqMessageConsumer;
@PreDestroy
public void destroyHook() {
try {
log.info("ShutDownHook destroy");
jmqMessageConsumer.pause();
fmqMessageConsumer.pause();
int i = 0;
while (i < waitTime) {
try {
Thread.sleep(1000);
log.info("距离服务关停还有{}秒", waitTime - i++);
} catch (Throwable e) {
log.error("异常", e);
}
}
} catch (Throwable e) {
log.error("异常", e);
}
}
}
•
When gracefully shutting down, the producer needs to be offline first, and a certain period of time must be reserved for consumption. There is a related stop.sh script for Xingyun deployment, which is implemented by writing methods in shutdown in the project.
jsf
jsf启停分析: see Jingdong internal cf document;
@Component
@Lazy(value = false)
public class ShutDown implements ApplicationContextAware {
private static Logger logger = LoggerFactory.getLogger(ShutDown.class);
@Value("${shutdown.waitTime:60}")
private int waitTime;
@Resource
com.jdjr.fmq.client.consumer.MessageConsumer fmqMessageConsumer;
@PostConstruct
public void init() {
logger.info("ShutDownHook init");
}
private ApplicationContext applicationContext = null;
@PreDestroy
public void destroyHook() {
try {
logger.info("ShutDownHook destroy");
destroyJsfProvider();
fmqMessageConsumer.pause();
int i = 0;
while (i < waitTime) {
try {
Thread.sleep(1000);
logger.info("距离服务关停还有{}秒", waitTime - i++);
} catch (Throwable e) {
logger.error("异常", e);
}
}
} catch (Throwable e) {
logger.error("异常", e);
}
}
private void destroyJsfProvider() {
logger.info("关闭所有JSF生产者");
if (null != applicationContext) {
String[] providerBeanNames = applicationContext.getBeanNamesForType(ProviderBean.class);
for (String name : providerBeanNames) {
try {
logger.info("尝试关闭JSF生产者" + name);
ProviderBean bean=(ProviderBean)applicationContext.getBean(name);
bean.destroy();
logger.info("关闭JSF生产者" + name + "成功");
} catch (BeanCreationNotAllowedException re){
logger.error("JSF生产者" + name + "未初始化，忽略");
} catch (Exception e) {
logger.error("关闭JSF生产者失败", e);
}
}
}
logger.info("所有JSF生产者已关闭");
}
@Override
public void setApplicationContext(ApplicationContext applicationContext) throws BeansException {
this.applicationContext = applicationContext;
((AbstractApplicationContext)applicationContext).registerShutdownHook();
}
}
•
The log cannot be printed due to the graceful shutdown of the application. After troubleshooting, the problem is found as follows: Through local debugging, it is found that the log printing thread of the graceful shutdown fails to print the actual countdown log.
absfactory-base-custcenter先销毁logback
<!-- fix-程序关停时，logback先销毁的问题-->
<context-param>
<param-name>logbackDisableServletContainerInitializer</param-name>
<param-value>true</param-value>
</context-param>
9. Summary
The existing springboot built-in Tomcat can achieve the effect of graceful shutdown through configuration parameters. However, because there are multiple technical cross-applications in the code in the business system, it does take time to study the underlying principles for different applications of Tomcat and springmvc to write related classes to achieve the same effect as springboot configuration parameter hosting.
Ministry of Industry and Information Technology: Do not provide network access services for unregistered apps Go 1.21 officially releases Linus to personally review the code, hoping to quell the "infighting" about the Bcachefs file system driver ByteDance launched a public DNS service 7-Zip official website was identified as a malicious website by Baidu Google releases AI code editor: Project IDX Tsinghua Report: Wenxin Yiyan firmly sits first in China, beyond the ChatGPT Vim project, the future meditation software will be launched, ChatGPT was founded by "China's first Linux person" with a daily cost of about 700,000 US dollars , OpenAI may be on the verge of bankruptcyAuthor: Jingdong Technology Song Huichao
Source: JD Cloud Developer Community
{{o.name}}
{{m.name}}