a background
C-side service application upgrades and restarts cause time-consuming instantaneous jitters, service timeouts, application monitoring alarms, and obvious upstream perception, resulting in poor user experience.
2. Reasons for jitter caused by application upgrade and restart
1. During the cold start phase of C-side service application upgrade and restart, it needs to reload and initialize various resources, such as database connections, cached data, etc., resulting in an instantaneous increase in time consumption.
2 After the application is restarted, the local cache becomes invalid and the application needs to reload data, resulting in longer time consumption.
3 After the application is restarted, a large number of requests accumulate, causing the system load to increase, and thread processing to be delayed, resulting in slow system response.
Three solutions
1. When stopping the service, first offline the JSF interface, and then upgrade the application. After the application is successfully started, the JSF service interface will be delayed and registered to the registration center.
2 JSF turns on the preheating function. After the service provider restarts, after JSF goes online, and after the consumer establishes a link with the service provider, the calling client first puts a small portion of the set proportion of traffic to the upgraded and restarted instance. During the preheating cycle within, gradually reaching 100%
3 In the application upgrade process orchestration, combine traffic replay, orchestrate the process, and then operate JSF online after traffic replay. The implementation is slightly complicated and relies on basic operation and maintenance.
Option 1 and Option 2 are simple to implement and have better preheating effects. They use the preheating function of JSF. The following comparison of effects comes from the JSF preheating function of Option 2.
4. Comparison of the effects of using JSF to enable the preheating function
1 Service provider configures warm-up period
Preheating initial weight: After the consumer connects to the provider, the provider's initial weight (the provider's weight will gradually return to the normal weight during the preheating period, usually 100)
2 The overall upgrade and restart monitoring of the preheated container instance is not enabled:
The application is deployed in 3 computer rooms, so we see 3 jitters. We can see that the computer room with the highest TP999 reached nearly 25 seconds.
The overall instance upgrade and restart effect after preheating is enabled:
The application is deployed in 3 computer rooms, so when we see 3 jitters, we can see that the computer room with the highest TP999 dropped to 2.5s, a decrease of 10 times.
3 Preheating is not enabled and single-instance application restart monitoring
max reached 20s, TP999 reached 18s
After preheating is enabled, the single-instance application restarts monitoring.
max has dropped to 2s, and TP999 will reach about 1.5s, a drop of 10 times. TPS shows linear growth within the configured warm-up period (60s)
Five things to note
Dependent service callers upgrade JSF to version 1.7.6 and above, and promote all callers to upgrade JSF
According to the startup time and characteristics of the application, the preheating period matching the application can be adjusted to achieve the best relative preheating effect. As for whether it is necessary to combine traffic playback and process orchestration to achieve preheating, you need to consider your own business conditions and implementation costs.
Author: Zhang Shilei of JD Technology
Source: JD Cloud Developer Community Please indicate the source when reprinting