foreword
The calculation of the health degree is time-consuming. The health degree of all applications needs to be calculated every minute according to the demand. When there are more applications, it is necessary to improve the performance and expand the computing power.
Improving performance can be handled by opening threads. Expanding computing power is nothing more than cluster deployment, and each node calculates a part. Here is an introduction to the research on computing solutions in App360.
Spring Scheduler
Introduction
Spring Scheduler is a timing task processing solution that comes with the Spring framework. It includes TaskExecutor and TaskScheduler, which are task execution and task scheduling respectively.
advantage
Seamlessly connected with Spring, easy to use and quick to get started.
shortcoming
The fatal disadvantage is that it will run at the same time in a cluster environment. If there are tasks that can only be executed on a certain host and cannot be controlled, it is best not to use them.
If you want to split the task to be executed on different machines, you can also execute it through database query sharding, etc., but once the node hangs, this part of the task will never be executed again.
Spring + Quartz
Introduction
Quartz is a very popular open source task scheduling framework, in which the quartz cluster can bring high availability and scalability to the scheduler through the functions of failover and load balancing.
Quartz is also the de facto standard for timed tasks in Java.
advantage
Through the database, configure the cluster mode. It can solve multiple hosts running the same task at the same time, and solve the fatal problem of Spring Schduler.
shortcoming
When there are multiple hosts in the cluster, and only one host is executing the task at the same time, only when the one executing the task hangs up will it switch to the other one.
If there are a lot of tasks that need to be executed periodically, the hardware resources are not effectively utilized, and in the final analysis, it is still single-point execution.
Lack of distributed parallel scheduling capabilities.
Spring Scheduler + Redis
Introduction
This is actually a solution proposed by the company itself. The scheduled execution still relies on the Spring Scheduler, but the information of each currently executed host and task is temporarily stored through Redis.
Then clean up the Redis staging data after execution.
advantage
Effective use of hardware resources ensures that each host in the cluster performs different tasks at the same time.
shortcoming
From design ideas to real implementation, there are many pits to go through. Although it is simple to implement, there are many issues that need to be considered.
For example: the first time the application is launched, all hosts will perform the same task at the same time. Redis node task information needs to be maintained by itself, and it is easy to generate multiple hosts in the cluster to perform the same tasks, etc.
XXL-JOB
Introduction
XXL-JOB is a lightweight distributed task scheduling framework, and its design idea adopts a central design.
advantage
Supports CRUD operations on tasks through web pages, supports dynamic modification of task status, suspend/resume tasks, and terminates running tasks, supports online configuration of scheduling task input parameters and online viewing of scheduling results.
Multi-node tasks will not be executed repeatedly, and its dynamic sharding function is also a highlight.
The solution is mature and has been tested by the production environment.
shortcoming
The database-based distributed function of Quartz is used, and many scheduling database tables need to be created, which is highly intrusive and too centralized.
More than a certain number of servers will cause a certain amount of pressure on the database.
Elastic-JOB
Introduction
Elastic-JOB is divided into Elastic-Job-Lite and Elastic-Job-Cloud, here we only talk about Elastic-Job-Lite.
Elastic-Job-Lite is Dangdang's open-source, decentralized and lightweight distributed task scheduling framework, which provides coordination services for distributed tasks in the form of jar packages.
It supports distributed scheduling coordination, elastic expansion and contraction, failover, re-triggering of missed execution jobs, parallel scheduling, self-diagnosis and repair, etc.
advantage
Rewrite the database-based distributed function of Quartz, and use Zookeeper to implement the registry instead of relying on the database.
The decentralized design ensures better horizontal elastic expansion.
Idempotency ensures that the running job task items will not be executed repeatedly in the cluster.
The solution is mature and has been tested by the production environment.
shortcoming
The dynamic sharding support is not good enough, and the sharding algorithm will only be triggered when the following three situations occur:
1. A new Job instance is added to the cluster;
2. An existing Job instance is offline (if the leader node is offline, the election will be performed first and then the execution of the sharding algorithm will be triggered);
3. Master node election.
Due to the decentralized design, it is not possible to add new jobs through the management console, nor to control the start and stop of the job process.
Summarize
Summarizing the above solutions, it is recommended to use mature solutions already on the market. Among them, XXL-JOB and Elastic-Job are more suitable.
Here is a comparison of the similarities and differences between the two:
Same point
Both are relatively mature distributed task scheduling solutions on the market, with complete technical documentation and a large user base. It can also meet the basic functional requirements of our timed tasks.
difference
The design concept of XXL-JOB is centralized, focusing on the simplicity of business implementation and the convenience of management, simple learning costs, and rich failure strategies and routing strategies.
It is recommended to use it in the scenario where the user base is relatively small and the number of servers is within a certain range.
The design of Elastic-Job is understood as decentralization, focusing on data, adding the idea of elastic expansion and data sharding, so as to maximize the utilization of distributed server resources.
However, the learning cost is relatively high, and it is recommended to be used when the amount of data is huge and the number of deployed servers is large.
At present, the first version of the asynchronous calculation of health is implemented for the company to specify the Spring Scheduler + Redis solution. However, considering that there may be many follow-up problems, the second version of the Elastic-JOB solution has also been implemented.
Of course, in addition to the scheduling schemes listed here, there are many other good frameworks.