News, open source lightweight dynamic thread pool monitoring - DynamicTp Join the Dromara community! welcome! !
The following is the introduction of DynamicTp by the authors of DynamicTp.
Hello everyone, today we are going to talk about a more practical topic, dynamic and monitorable thread pool practice. The address of the new open source project (DynamicTp) is at the end of the article. Welcome to exchange and learn.
background
Do you have the following pain points when using ThreadPoolExecutor?
1. A ThreadPoolExecutor is created in the code, but I don't know how many core parameter settings are appropriate
2. Set parameter values based on experience. After going online, it is found that you need to adjust and change the code to restart the service, which is very troublesome.
3. The thread pool is a black box for developers, and the running situation cannot be sensed until a problem occurs
If you have the above pain points, the dynamic monitorable thread pool (DynamicTp) may be able to help you.
If you have seen the source code of ThreadPoolExecutor, you can probably know that it actually provides some set methods, which can dynamically modify the corresponding values at runtime. These methods are:
public void setCorePoolSize(int corePoolSize);
public void setMaximumPoolSize(int maximumPoolSize);
public void setKeepAliveTime(long time, TimeUnit unit);
public void setThreadFactory(ThreadFactory threadFactory);
public void setRejectedExecutionHandler(RejectedExecutionHandler handler);
Nowadays, most Internet projects are actually deployed as microservices and have their own service governance system. The distributed configuration center in the microservice component plays the role of dynamically modifying the configuration and taking effect in real time. So can we combine the configuration center to dynamically adjust the parameters of the thread pool at runtime? The answer is yes, and the configuration center is relatively highly available. Using it, you don’t have to worry too much about configuration push problems, and it can also reduce the difficulty and workload of developing dynamic thread pool components.
In summary, we summarize the following background
-
Extensive: In Java development, if you want to improve system performance, thread pool is already a basic tool that more than 90% of people will choose to use
-
Uncertainty: Many thread pools may be created in the project, both IO-intensive and CPU-intensive, but the parameters of the thread pool are not well determined; a mechanism is required to dynamically adjust the parameters during operation
-
Non-perceptual, various indicators during the running process of the thread pool are generally not perceptible; a set of monitoring and alarm mechanisms are required to allow developers to perceive the running status of the thread pool before and during the event, and deal with it in time
-
High availability, configuration changes need to be pushed to the client in time; a high-availability configuration management push service is required. The configuration center is a component used by most Internet systems now, and its combination can greatly reduce the amount of development and access difficulty.
Introduction
We make some extensions to the thread pool ThreadPoolExecutor based on the configuration center to realize the dynamic modification of the running thread pool parameters, which will take effect in real time; and monitor the running status of the thread pool in real time, trigger the alarm policy when the alarm policy is set, and the alarm information will be pushed to the office platform ( Dingding, Qiwei, etc.). The alarm dimensions include (queue capacity, thread pool activity, refusal to trigger, task timeout, etc.); at the same time, thread pool indicator data is also collected regularly for visual use by the monitoring platform. This enables us to perceive the load of the thread pool at all times, and adjust it in time according to the situation, so as to avoid problems that affect the online business.
| __ \ (_) |__ __|
| | | |_ _ _ __ __ _ _ __ ___ _ ___| |_ __
| | | | | | | '_ \ / _` | '_ ` _ | |/ __| | '_ \
| |__| | |_| | | | | (_| | | | | | | | (__| | |_) |
|_____/ __, |_| |_|__,_|_| |_| |_|_|___|_| .__/
__/ | | |
|___/ |_|
:: Dynamic Thread Pool ::
characteristic
-
Refer to Meituan thread pool practice , dynamically manage thread pool parameters, and add monitoring and alarm functions
-
Based on the Spring framework, it only supports the use of SpringBoot projects. It is lightweight and can be used by introducing starter.
-
Dynamic adjustment of thread pool parameters based on the configuration center, which takes effect in real time; integrated with the mainstream configuration center, already supports Nacos, Apollo, Zookeeper, Consul, and also provides SPI interface for custom extension implementation
-
Built-in notification alarm function, providing multiple alarm dimensions (configuration change notification, activity alarm, capacity threshold alarm, rejection trigger alarm, task execution or waiting timeout alarm), default support for enterprise WeChat, DingTalk alarm, and SPI interface is provided to customize Extended implementation
-
The built-in thread pool indicator collection function supports three methods: MicroMeter, JsonLog log output, and Endpoint, which can be implemented by custom extension through the SPI interface
-
Integrate and manage the thread pools of common third-party components, and integrate the thread pool management of SpringBoot's built-in WebServer (Tomcat, Undertow, Jetty)
-
Provide task wrapping function, just implement TaskWrapper interface, such as TtlTaskWrapper can support thread pool context information transfer
-
The JUC common thread pool can also be monitored by the framework (@DynamicTp); refer to the Tomcat thread pool to provide EagerDtpExecutor for io-intensive scenarios
Architecture design
Mainly divided into four modules
-
Configuration change monitoring module:
1. Monitor the specified configuration files of a specific configuration center (Nacos, Apollo, Zookeeper, Consul have been implemented), and other implementations can be extended through the internally provided SPI interface
2. Parse the content of the configuration file, built-in parsing of yml and properties configuration files, and can extend other implementations through the SPI interface provided internally
3. Notify the thread pool management module to achieve refresh
-
Thread pool management module:
1. Pull the configuration from the configuration center when the service starts, generate a thread pool instance and register it in the internal thread pool registry and the Spring container
2. When the monitoring module monitors the configuration change, it transmits the change information to the management module to refresh the thread pool parameters
3. Obtain the thread pool instance according to the thread pool name through dependency injection (recommended) or the getExecutor() method in the code
-
Monitoring module:
To realize the collection and output of monitoring indicators, the following three methods are provided by default, and other implementations can also be extended through the internally provided SPI interface
1. By default, JsonLog is output to disk, and you can collect and analyze logs by yourself, store and display
2. MicroMeter collection, introduction of MicroMeter related dependencies, and exposure of related endpoints
3. Thunderstorm custom Endpoint endpoint, which can be accessed in real time through http
-
Notification alarm module:
Connect to the office platform to realize the notification and alarm function. By default, Dingding and Qiwei are implemented. Other implementations can be extended through the SPI interface provided internally. The types of notification and alarms are as follows
1. Thread pool main parameter change notification
2. The blocking queue capacity reaches the set alarm threshold
3. The thread pool activity reaches the set alarm threshold
4. Trigger rejection policy alarm, format: A/B, A: the cumulative number of the two alarm intervals before and after the alarm item, B: the cumulative total of the alarm item
5. Task execution timeout alarm, format: A/B, A: the cumulative number of the two alarm intervals before and after the alarm item, B: the cumulative total of the alarm item
6. Task waiting timeout alarm, format: A/B, A: the cumulative number of the two alarm intervals before and after the alarm item, B: the cumulative total of the alarm item
use
1. Introduce the dependency of the corresponding configuration center
2. Configure the thread pool instance in the configuration center. For the configuration, refer to the following (all configuration items are given, and the configuration items have default values)
3. Add the @EnableDynamicTp annotation to the startup class
4. Use @Resource or @Autowired annotation to inject, or get through DtpRegistry.getExecutor("name")
5. If the ordinary JUC thread pool wants to be monitored, you can add @DynamicTp annotation when @Bean is defined
6.tips: When the dynamic thread pool instance service is started, it will be dynamically registered in the Spring container according to the configuration of the configuration center. It is recommended not to use @Bean programmatically to repeatedly declare the same thread pool instance, and configure it directly in the configuration center.
7. Refer to the Example example for details
-
Thread pool configuration (yml type)
spring: dynamic: tp: enabled: true enabledBanner: true # 是否开启banner打印，默认true enabledCollect: false # 是否开启监控指标采集，默认false collectorType: logging # 监控数据采集器类型（JsonLog | MicroMeter），默认logging logPath: /home/logs # 监控日志数据路径，默认 ${user.home}/logs monitorInterval: 5 # 监控时间间隔（报警判断、指标采集），默认5s platforms: # 通知报警平台配置 - platform: wechat urlKey: 3a7500-1287-4bd-a798-c5c3d8b69c # 替换 receivers: test1,test2 # 接受人企微名称 - platform: ding urlKey: f80dad441fcd655438f4a08dcd6a # 替换 secret: SECb5441fa6f375d5b9d21 # 替换，非sign模式可以没有此值 receivers: 15810119805 # 钉钉账号手机号 tomcatTp: # tomcat web server线程池配置 minSpare: 100 max: 400 jettyTp: # jetty web server线程池配置 min: 100 max: 400 undertowTp: # undertow web server线程池配置 coreWorkerThreads: 100 # 核心线程数 maxWorkerThreads: 400 # 最大线程数 workerKeepAlive: 40 executors: # 动态线程池配置，都有默认值，采用默认值的可以不配置该项，减少配置量 - threadPoolName: dtpExecutor1 executorType: common # 线程池类型common、eager：适用于io密集型 corePoolSize: 6 maximumPoolSize: 8 queueCapacity: 200 queueType: VariableLinkedBlockingQueue # 任务队列，查看源码QueueTypeEnum枚举类 rejectedHandlerType: CallerRunsPolicy # 拒绝策略，查看RejectedTypeEnum枚举类 keepAliveTime: 50 allowCoreThreadTimeOut: false threadNamePrefix: test # 线程名前缀 waitForTasksToCompleteOnShutdown: false # 参考spring线程池设计 awaitTerminationSeconds: 5 # 单位（s） preStartAllCoreThreads: false # 是否预热核心线程，默认false runTimeout: 200 # 任务执行超时阈值，目前只做告警用，单位（ms） queueTimeout: 100 # 任务在队列等待超时阈值，目前只做告警用，单位（ms） taskWrapperNames: ["ttl"] # 任务包装器名称，集成TaskWrapper接口 notifyItems: # 报警项，不配置自动会按默认值配置（变更通知、容量报警、活性报警、拒绝报警、任务超时报警） - type: capacity # 报警项类型，查看源码 NotifyTypeEnum枚举类 enabled: true threshold: 80 # 报警阈值 platforms: [ding,wechat] # 可选配置，不配置默认拿上层platforms配置的所以平台 interval: 120 # 报警间隔（单位：s）
-
Code call, obtained from DtpRegistry according to the thread pool name, or through dependency injection (recommended, more elegant)
@Resource private ThreadPoolExecutor dtpExecutor1; public void exec() { dtpExecutor1.execute(() -> System.out.println("test")); }public static void main(String[] args) { DtpExecutor dtpExecutor = DtpRegistry.getExecutor("dtpExecutor1"); dtpExecutor.execute(() -> System.out.println("test")); } -
Detailed use case reference
exampleproject
notification alarm
-
Triggering the alarm threshold will push the corresponding alarm message (liveness, capacity, reject, timeout) and the corresponding field will be highlighted
-
Configuration changes will push notification messages and the changed fields will be highlighted
monitor
Monitoring data
Configure the monitoring indicator collection type through the collectType property, the default is logging
-
MicroMeter: Collect to the corresponding platform by introducing related MicroMeter dependencies (such as Prometheus, InfluxDb...)
-
Logging: Periodically collect indicator data and output it to disk in Json log format, address {appName}.monitor.log
-
Expose EndPoint endpoint (dynamic-tp), which can be requested through http
[ { "dtp_name": "remoting-call", "core_pool_size": 6, "maximum_pool_size": 12, "queue_type": "SynchronousQueue", "queue_capacity": 0, "queue_size": 0, "fair": false, "queue_remaining_capacity": 0, "active_count": 0, "task_count": 21760, "completed_task_count": 21760, "largest_pool_size": 12, "pool_size": 6, "wait_task_count": 0, "reject_count": 124662, "reject_handler_name": "CallerRunsPolicy" } ]
Follow the project
If you have any ideas or suggestions for the project, you can create issues and improve the project together
Warehouse address, welcome to like the project!
https://github.com/dromara/dynamic-tp
https://gitee.com/dromara/dynamic-tp