Today, let's talk about a more practical topic, dynamic monitorable and observable thread pool practice.
This is a brand new open source project. The author provides a very good idea to solve the observable problem of thread pools.
This open source project is called: DynamicTp
The address is at the end of the article.
write in front
Friends with a little Java programming experience know that the essence of Java is in the juc package, which is the masterpiece of the famous old man Doug Lea. To evaluate a programmer's Java level, to a certain extent, it depends on his mastery of some technologies under the juc package. How is it? This is also one of the basic technical points that must be asked in the interview.
The juc package mainly includes:
1. Atomic class (AtomicXXX)
2. Lock class (XXXLock)
3. Thread synchronization class (AQS, CountDownLatch, CyclicBarrier, Semaphore, Exchanger)
4. Task executor class (Executor system class, including today's protagonist ThreadPoolExecutor)
5. Concurrent collection classes (ConcurrentXXX, CopyOnWriteXXX) related collection classes
6. Blocking queue class (BlockingQueue inheritance system class)
7.Future related classes
8. Some other auxiliary tools
In the multi-threaded programming scenario, these classes are essential skills. Knowing these can help us write high-quality, high-performance, and bug-free code. At the same time, these are some of the more difficult technologies in Java, and you need to persevere and apply what you have learned. , feel the mystery brought by them in use.
The above briefly lists the functional classification of the juc package. In this article, we mainly introduce the dynamic monitorable thread pool, so the specific content will not be discussed. I will talk about it when I have time in the future. Before reading this article, I hope that readers should have some experience in using thread pool ThreadPoolExecutor, otherwise it will look a bit confusing.
If you are not very familiar with ThreadPoolExecutor, it is recommended to read the following two articles
javadoop： https://www.javadoop.com/post/java-thread-pool
Meituan Technology Blog: https://tech.meituan.com/2020/04/02/java-pooling-pratice-in-meituan.html
background
Do you have the following pain points when using ThreadPoolExecutor?
1. A ThreadPoolExecutor is created in the code, but I don't know how many core parameter settings are appropriate
2. Set parameter values based on experience. After going online, it is found that you need to adjust and change the code to restart the service, which is very troublesome.
3. The thread pool is a black box for developers, and the running situation cannot be sensed until a problem occurs
If you have the above pain points, the dynamic monitorable thread pool (DynamicTp) introduced in this article may be able to help you.
If you have seen the source code of ThreadPoolExecutor, you can probably know that it actually provides some set methods, which can dynamically modify the corresponding values at runtime. These methods are:
public void setCorePoolSize(int corePoolSize);
public void setMaximumPoolSize(int maximumPoolSize);
public void setKeepAliveTime(long time, TimeUnit unit);
public void setThreadFactory(ThreadFactory threadFactory);
public void setRejectedExecutionHandler(RejectedExecutionHandler handler);
Nowadays, most Internet projects are actually deployed as microservices and have their own service governance system. The distributed configuration center in the microservice component plays the role of dynamically modifying the configuration and taking effect in real time. So can we combine the configuration center to dynamically adjust the parameters of the thread pool at runtime? The answer is yes, and the configuration center is relatively highly available. Using it, you don’t have to worry too much about configuration push problems, and it can also reduce the difficulty and workload of developing dynamic thread pool components.
In summary, we summarize the following background
- Extensive: In Java development, if you want to improve system performance, thread pool is already a basic tool that more than 90% of people choose to use
- Uncertainty: Many thread pools may be created in the project, both IO-intensive and CPU-intensive, but the parameters of the thread pool are not easy to determine; a mechanism is required to dynamically adjust the parameters during operation
- Non-perceptual, various indicators during the running process of the thread pool are generally not perceptible; a set of monitoring and alarm mechanisms are required to allow developers to perceive the running status of the thread pool before and during the event, and deal with it in time
- High availability, configuration changes need to be pushed to the client in time; a high-availability configuration management push service is required. The configuration center is a component used by most Internet systems now, and its combination can greatly reduce the amount of development and access difficulty.
Introduction
We make some extensions to the thread pool ThreadPoolExecutor based on the configuration center to realize the dynamic modification of the running thread pool parameters, which will take effect in real time; and monitor the running status of the thread pool in real time, trigger the alarm policy when the alarm policy is set, and the alarm information will be pushed to the office platform ( Dingding, Qiwei, etc.). The alarm dimensions include (queue capacity, thread pool activity, refusal to trigger, etc.); at the same time, thread pool indicator data is also collected regularly for the visualization of the monitoring platform. This enables us to perceive the load of the thread pool at all times, and adjust it in time according to the situation, so as to avoid problems that affect the online business.
| __ \ (_) |__ __|
| | | |_ _ _ __ __ _ _ __ ___ _ ___| |_ __
| | | | | | | '_ \ / _` | '_ ` _ | |/ __| | '_ \
| |__| | |_| | | | | (_| | | | | | | | (__| | |_) |
|_____/ __, |_| |_|__,_|_| |_| |_|_|___|_| .__/
__/ | | |
|___/ |_|
:: Dynamic Thread Pool ::
characteristic
- Refer to Meituan thread pool practice, dynamically manage thread pool parameters, and add monitoring and alarm functions
- Based on the Spring framework, it only supports the use of SpringBoot projects. It is lightweight and can be eaten by introducing starter.
- Dynamic adjustment of thread pool parameters based on the configuration center, which takes effect in real time; integrated with the mainstream configuration center, supports Nacos and Apollo by default, and also provides SPI interface that can be customized and extended.
- Built-in notification and alarm function, providing a variety of alarm dimensions (configuration change notification, activity alarm, capacity threshold alarm, rejection policy trigger alarm), default support for enterprise WeChat, DingTalk alarm, and SPI interface for custom expansion implementation
- The built-in thread pool indicator collection function supports three methods: MicroMeter, JsonLog log output, and Endpoint, which can be implemented by custom extension through the SPI interface
Architecture design
Mainly divided into four modules
-
Configuration change monitoring module:
1. Monitor the specified configuration file of a specific configuration center (Nacos, Apollo are implemented by default), and other implementations can be extended through the SPI interface provided internally
2. Parse the content of the configuration file, built-in parsing of yml and properties configuration files, and can extend other implementations through the SPI interface provided internally
3. Notify the thread pool management module to achieve refresh
-
Thread pool management module:
1. When the service starts, the configuration information is pulled from the configuration center, and the generated thread pool instance is registered in the internal thread pool registry
2. When the monitoring module monitors the configuration change, it transmits the change information to the management module to refresh the thread pool parameters
3. In the code, the getExecutor() method is used to obtain the thread pool object instance according to the thread pool name
-
Monitoring module:
Realize the collection and output of monitoring indicators. The following three methods are provided by default, and other implementations can also be extended through the SPI interface provided internally.
1. The default implementation of Json log output to disk
2. MicroMeter collection, introduction of MicroMeter related dependencies
3. Thunder Endpoint endpoint, which can be accessed through http
-
Notification alarm module:
It connects to the office platform and realizes the notification and alarm function. By default, Dingding and Qiwei are implemented. Other implementations can be extended through the SPI interface provided internally. The types of notification alarms are as follows
1. Thread pool parameter change notification
2. The blocking queue capacity reaches the set threshold alarm
3. Thread pool activity reaches the set threshold alarm
4. Trigger a rejection policy alert
use
-
maven dependencies
<dependency> <groupId>io.github.lyh200</groupId> <artifactId>dynamic-tp-spring-cloud-starter</artifactId> <version>1.0.2-RELEASE</version> </dependency> -
thread pool configuration
spring: dynamic: tp: enabled: true enabledBanner: true # 是否开启banner打印，默认true enabledCollect: false # 是否开启监控指标采集，默认false collectorType: logging # 监控数据采集器类型（JsonLog | MicroMeter），默认logging logPath: /home/logs # 监控日志数据路径，默认${user.home}/logs monitorInterval: 5 # 监控时间间隔（报警判断、指标采集），默认5s nacos: # nacos配置，不配置有默认值（规则name-dev.yml这样） dataId: dynamic-tp-demo-dev.yml group: DEFAULT_GROUP apollo: # apollo配置，不配置默认拿apollo配置第一个namespace namespace: dynamic-tp-demo-dev.yml configType: yml # 配置文件类型 platforms: # 通知报警平台配置 - platform: wechat urlKey: 3a7500-1287-4bd-a798-c5c3d8b69c # 替换 receivers: test1,test2 # 接受人企微名称 - platform: ding urlKey: f80dad441fcd655438f4a08dcd6a # 替换 secret: SECb5441fa6f375d5b9d21 # 替换，非sign模式可以没有此值 receivers: 15810119805 # 钉钉账号手机号 executors: # 动态线程池配置 - threadPoolName: dynamic-tp-test-1 corePoolSize: 6 maximumPoolSize: 8 queueCapacity: 200 queueType: VariableLinkedBlockingQueue # 任务队列，查看源码QueueTypeEnum枚举类 rejectedHandlerType: CallerRunsPolicy # 拒绝策略，查看RejectedTypeEnum枚举类 keepAliveTime: 50 allowCoreThreadTimeOut: false threadNamePrefix: test # 线程名前缀 notifyItems: # 报警项，不配置自动会配置（变更通知、容量报警、活性报警、拒绝报警） - type: capacity # 报警项类型，查看源码 NotifyTypeEnum枚举类 enabled: true threshold: 80 # 报警阈值 platforms: [ding,wechat] # 可选配置，不配置默认拿上层platforms配置的所以平台 interval: 120 # 报警间隔（单位：s） - type: change enabled: true - type: liveness enabled: true threshold: 80 - type: reject enabled: true threshold: 1 -
Code generation, service startup will be automatically registered
@Configuration public class DtpConfig { @Bean public DtpExecutor demo1Executor() { return DtpCreator.createDynamicFast("demo1-executor"); } @Bean public ThreadPoolExecutor demo2Executor() { return ThreadPoolBuilder.newBuilder() .threadPoolName("demo2-executor") .corePoolSize(8) .maximumPoolSize(16) .keepAliveTime(50) .allowCoreThreadTimeOut(true) .workQueue(QueueTypeEnum.SYNCHRONOUS_QUEUE.getName(), null, false) .rejectedExecutionHandler(RejectedTypeEnum.CALLER_RUNS_POLICY.getName()) .buildDynamic(); } } -
Code call, obtained according to the thread pool name
public static void main(String[] args) { DtpExecutor dtpExecutor = DtpRegistry.getExecutor("dynamic-tp-test-1"); dtpExecutor.execute(() -> System.out.println("test")); }
Precautions
-
Parameters configured in the configuration file override parameters configured through code generation
-
Only the VariableLinkedBlockingQueue type of the blocking queue can modify the capacity. The function of this type is similar to that of the LinkedBlockingQueue, except that the capacity is not a final type and can be modified. VariableLinkedBlockingQueue refers to the implementation of RabbitMq
-
Start and see the following log output to prove that the access is successful
| __ \ (_) |__ __| | | | |_ _ _ __ __ _ _ __ ___ _ ___| |_ __ | | | | | | | '_ \ / _` | '_ ` _ | |/ __| | '_ \ | |__| | |_| | | | | (_| | | | | | | | (__| | |_) | |_____/ __, |_| |_|__,_|_| |_| |_|_|___|_| .__/ __/ | | | |___/ |_| :: Dynamic Thread Pool :: DynamicTp register, executor: DtpMainPropWrapper(dtpName=dynamic-tp-test-1, corePoolSize=6, maxPoolSize=8, keepAliveTime=50, queueType=VariableLinkedBlockingQueue, queueCapacity=200, rejectType=RejectedCountableCallerRunsPolicy, allowCoreThreadTimeOut=false) -
Configuration changes will push notification messages and the changed fields will be highlighted
DynamicTp [dynamic-tp-test-1] refresh end, changed keys: [corePoolSize, queueCapacity], corePoolSize: [6 => 4], maxPoolSize: [8 => 8], queueType: [VariableLinkedBlockingQueue => VariableLinkedBlockingQueue], queueCapacity: [200 => 2000], keepAliveTime: [50s => 50s], rejectedType: [CallerRunsPolicy => CallerRunsPolicy], allowsCoreThreadTimeOut: [false => false]
notification alarm
Triggering the alarm threshold will push the corresponding alarm message (liveness, capacity, reject) and the corresponding field will be highlighted
Configuration changes will push notification messages and the changed fields will be highlighted
Monitoring logs
Configure the monitoring indicator collection type through the collectType property, the default is logging
-
MicroMeter: By introducing relevant MicroMeter dependencies to the corresponding platform (such as Prometheus, InfluxDb...)
-
Logging: Periodically collect indicator data and output it to disk in Json log format, address ${logPath}/dy namictp/${appName}.monitor.log
2022-01-11 00:25:20.599 INFO [dtp-monitor-thread-1:d.m.log] {"activeCount":0,"queueSize":0,"largestPoolSize":0,"poolSize":0,"rejectHandlerName":"RejectedCountableCallerRunsPolicy","queueCapacity":1024,"fair":false,"rejectCount":0,"waitTaskCount":0,"taskCount":0,"queueRemainingCapacity":1024,"corePoolSize":6,"queueType":"VariableLinkedBlockingQueue","completedTaskCount":0,"dtpName":"remoting-call","maximumPoolSize":8} 2022-01-11 00:25:25.603 INFO [dtp-monitor-thread-1:d.m.log] {"activeCount":0,"queueSize":0,"largestPoolSize":0,"poolSize":0,"rejectHandlerName":"RejectedCountableCallerRunsPolicy","queueCapacity":1024,"fair":false,"rejectCount":0,"waitTaskCount":0,"taskCount":0,"queueRemainingCapacity":1024,"corePoolSize":6,"queueType":"VariableLinkedBlockingQueue","completedTaskCount":0,"dtpName":"remoting-call","maximumPoolSize":8} 2022-01-11 00:25:30.609 INFO [dtp-monitor-thread-1:d.m.log] {"activeCount":0,"queueSize":0,"largestPoolSize":0,"poolSize":0,"rejectHandlerName":"RejectedCountableCallerRunsPolicy","queueCapacity":1024,"fair":false,"rejectCount":0,"waitTaskCount":0,"taskCount":0,"queueRemainingCapacity":1024,"corePoolSize":6,"queueType":"VariableLinkedBlockingQueue","completedTaskCount":0,"dtpName":"remoting-call","maximumPoolSize":8} 2022-01-11 00:25:35.613 INFO [dtp-monitor-thread-1:d.m.log] {"activeCount":0,"queueSize":0,"largestPoolSize":0,"poolSize":0,"rejectHandlerName":"RejectedCountableCallerRunsPolicy","queueCapacity":1024,"fair":false,"rejectCount":0,"waitTaskCount":0,"taskCount":0,"queueRemainingCapacity":1024,"corePoolSize":6,"queueType":"VariableLinkedBlockingQueue","completedTaskCount":0,"dtpName":"remoting-call","maximumPoolSize":8} 2022-01-11 00:25:40.616 INFO [dtp-monitor-thread-1:d.m.log] {"activeCount":0,"queueSize":0,"largestPoolSize":0,"poolSize":0,"rejectHandlerName":"RejectedCountableCallerRunsPolicy","queueCapacity":1024,"fair":false,"rejectCount":0,"waitTaskCount":0,"taskCount":0,"queueRemainingCapacity":1024,"corePoolSize":6,"queueType":"VariableLinkedBlockingQueue","completedTaskCount":0,"dtpName":"remoting-call","maximumPoolSize":8} -
Expose EndPoint endpoint (dynamic-tp), which can be requested through http
[ { "dtp_name": "remoting-call", "core_pool_size": 6, "maximum_pool_size": 12, "queue_type": "SynchronousQueue", "queue_capacity": 0, "queue_size": 0, "fair": false, "queue_remaining_capacity": 0, "active_count": 0, "task_count": 21760, "completed_task_count": 21760, "largest_pool_size": 12, "pool_size": 6, "wait_task_count": 0, "reject_count": 124662, "reject_handler_name": "CallerRunsPolicy" }, { "max_memory": "228 MB", "total_memory": "147 MB", "free_memory": "44.07 MB", "usable_memory": "125.07 MB" } ]
project address
gitee address: https://gitee.com/yanhom/dynamic-tp-spring-cloud-starter
github address : https://github.com/lyh200/dynamic-tp-spring-cloud-starter
Contact the author
If you have any ideas or suggestions for the project, you can add it to the author's WeChat in the above address for communication, or create issues and improve the project together!
Finally, if you support it, I hope you can click a star.
