1. Why have I never heard of this product?
Friends who often use Quartz or Spring Task will more or less encounter several pain points, such as:
1. Don’t dare to follow the multi-node deployment of application services easily, and may cause system logic errors due to repeated execution;
2. , Quartz's cluster is only used for HA, the increase in the number of nodes does not improve our execution efficiency each time, that is, it cannot achieve horizontal expansion;
In Dangdang's ddframe framework, a task scheduling system is needed. There are two ways to realize it, one is to modify open source products, and the other is to build based on open source products, that is, packaging. Dangdang chose the latter. At first, the scheduling system was called dd-job. It is a decentralized distributed scheduling framework. Because the database lacks distributed coordination functions (such as master election), after replacing it with Zookeeper, the functions of elastic expansion and data fragmentation are added. Elastic-Job is a job framework separated from the dd-job job module in ddframe. It is developed based on Quartz and Curator and was open sourced in 2015.
Elastic-Job is a Java distributed timing task developed and open source based on Zookepper and Quartz by Dangdang architects Zhang Liang, Cao Hao and Jiang Shujian (it has nothing to do with the famous ElasticSearch), which solves the disadvantage that Quartz does not support distributed . The main functions of Elastic-Job include support for elastic expansion, centralized management and monitoring of jobs through Zookepper, and support for failover. At the beginning, there was only one elastic-job-core project. After version 2.X, it is mainly divided into two sub-projects, Elastic-Job-Lite and Elastic-Job-Cloud. Among them, Elastic-Job-Lite is positioned as a lightweight decentralized solution, providing coordination services for distributed tasks in the form of jar packages. Elastic-Job-Cloud uses the Mesos + Docker solution to provide additional services such as resource management, application distribution, and process isolation (the difference from Lite is only the deployment method, using the same API, only need to develop once).
The reason why I feel that this product is not used by anyone is because after the project was donated to Apache in 2020, the original programming methods were basically deleted, and now it is difficult to find native programming methods on the Internet. What's even more outrageous is that the demo given by the official website still misses a lot, that is to say, if you follow the official website to write it down step by step, the result is that you can't build it...
Then why do I still have to write such an article? It is reasonable to say that a thing that even the official website has given up, why spend all this energy. Because there is a key concept in Elastic-Job: sharding, that is, the task sharding strategy, which is the biggest difference between it and Quartz. Now Elastic-Job has become a second-level sub-project, and its registration center depends on Zookeeper, so if you want to use Elastic-Job, you must first install the Zookeeper service.
2. Problems left by Quartz
① Assuming there are multiple running nodes in a Quartz cluster, how do you decide which tasks run on which nodes? Quartz's processing method is very simple and rude, it is random. Use the information stored in the database to seize the task authority bound to the next trigger that is about to be triggered, and does not support the coordination of task execution nodes;
② When dealing with a very complex task, the performance of a certain node is always limited. If a complex task can be split into multiple sub-tasks, which can be handed over to different nodes for collaborative processing, the efficiency will be twice the result with half the effort;
③ Quartz itself does not support graphical management pages, which is very inconvenient for task management;
3. First try Elastic-Job
Elastic-Job can solve all the above problems. Elastic-Job is serious about making up for the shortcomings of Quartz, but it must be admitted that it is really not particularly useful. Unfortunately, the official website of Elastic-Job is no longer accessible. This official website was bought to Kuaicheng (a VPN tool) a few years ago, and now its official website has been reduced to a secondary website under shardingsphere: https: //shardingsphere.apache.org/elasticjob/index_zh.html
Start the Zookeeper service
Because Elastic-Job depends on Zookeeper, first ensure that the related Zookeeper service starts successfully. It is convenient to demonstrate here, so I will no longer set up a virtual machine to set up a Zookeeper service, but directly start the Zookeeper service locally. There are a few points to note:
1. After decompressing the compressed package of Zookeeper, enter the directory, create a new data directory to store data, and create a new logs directory to store logs;
2. Enter the conf directory, copy a copy of zoo_sample.cfg, and name it zoo.cfg as the Zookeeper configuration file, and some basic modifications are required;
3. Enter the bin directory in Zookeeper and execute: zkServer.cmd
If you see the log output ZooKeeper audit is enabled., it means that the ZooKeeper service starts successfully.
Introducing Elastic-Job related dependencies
<dependency>
<groupId>org.apache.shardingsphere.elasticjob</groupId>
<artifactId>elasticjob-lite-core</artifactId>
<version>3.0.1</version>
</dependency>At present, the latest version of Maven Central Warehouse is 3.0.2, updated on October 23, 22, and 3.0.1 is currently the most used version, updated on October 11, 21. Good guy, it's updated once a year. Looking at the frequency of this maintenance, you can tell that this thing is useless.
SimpleJob
Customize the tasks to be executed and implement the SimpleJob interface
Build registry, configure task job
Start the Elastic-Job service
It can be seen that the specified 7 shards execute the task. Here is just a simple print sentence. In actual business, you can judge and control different shards to execute different tasks through code judgment, and then you can split and execute tasks.
The tutorial on the official website doesn't even give how to build a job configuration.....
DataFlowJob
The above demonstration is the simplest task mode. Elastic-Job also provides another data flow mode: DataFlowJob, which is used to process data flow. The fetchData() and processData() methods must be implemented, one is used to obtain data, the other is used to process the obtained data, and the others are no different from SimpleJob.
Implement the DataFlowJob interface
Build registry, configure task job
Start the Elastic-Job service
As you can see, the three defined shards perform task execution
ScriptJob
Script-type job means script-type job, which supports all types of scripts such as shell, python, and perl. To put it bluntly, this type of job is to execute script files regularly. Write the script file that needs to be executed, tell it where it is, what its name is, and it will automatically run for you when it is clicked.
When using the ScriptJob type, there are a few points to note:
1. When building the Scheduler, the parameter elasticJobType is fixed as "SCRIPT";
2. When building the job configuration, the first one of the .setProperty method The parameter is also fixed "script.command.line", and the second parameter is the absolute path of the script file;
The content of the script is a very simple sentence: @echo ------[script task] Sharding Context: %*
4. Realize custom job types through SPI
Customize the task interface, inherit the ElasticJob interface
Implement a custom task interface
Customize the task executor interface, inheriting the ClassedJobItemExecutor interface
Implement a custom task executor interface
Create configuration file
In the resources.META-INF.services directory, create a configuration file with the full path name of the task executor interface, and the content is the full path name of the custom task executor implementation class
Execute custom jobs
package com.feenix.elasticjob.service;
import com.feenix.elasticjob.service.impl.FeenixJobImpl;
import org.apache.shardingsphere.elasticjob.api.JobConfiguration;
import org.apache.shardingsphere.elasticjob.lite.api.bootstrap.impl.ScheduleJobBootstrap;
import org.apache.shardingsphere.elasticjob.reg.base.CoordinatorRegistryCenter;
import org.apache.shardingsphere.elasticjob.reg.zookeeper.ZookeeperConfiguration;
import org.apache.shardingsphere.elasticjob.reg.zookeeper.ZookeeperRegistryCenter;
public class Application {
public static void main(String[] args) {
new ScheduleJobBootstrap(createRegistryCenter(),
new FeenixJobImpl(),
createJobConfiguration())
.schedule();
}
// 构建Zookeeper注册中心
private static CoordinatorRegistryCenter createRegistryCenter() {
ZookeeperConfiguration zkConfiguration = new ZookeeperConfiguration("192.168.0.31:2181",
"feenix-job");
zkConfiguration.setConnectionTimeoutMilliseconds(100000);
zkConfiguration.setMaxRetries(10);
ZookeeperRegistryCenter zkRegistryCenter = new ZookeeperRegistryCenter(zkConfiguration);
zkRegistryCenter.init();
return zkRegistryCenter;
}
// 作业配置
private static JobConfiguration createJobConfiguration() {
String jobs = "0=宋远桥,1=俞莲舟,2=俞岱岩,3=张松溪,4=张翠山,5=殷梨亭,6=莫声谷";
return JobConfiguration.newBuilder("FeenixJobImpl", 7)
.cron("0/3 * * * * ?")
.shardingItemParameters(jobs)
// 使用自定义的作业分片策略
/*.jobSharding StrategyType("Shuffle")*/
// 允许客户端配置覆盖注册中心
.overwrite(true)
// 故障转移
.failover(true)
.build();
}
}
5. Fragmentation
The concept of task sharding in Elastic-Job enables tasks to run in a distributed environment, and each task server only runs the shards assigned to the server. With the increase or downtime of servers, Elastic-Job will perceive the changes in the number of servers in near real time, so as to re-allocate more reasonable task fragmentation items for distributed servers. Yes, tasks can improve efficiency as resources increase.
However, as mentioned above, Elastic-Job does not directly provide data processing functions, but distributes the shard items to each running job server. Developers need to handle the correspondence between shard items and businesses by themselves. Shard items are numbers, starting from 0. For example: split the database according to the regional level, database A is the data of Beijing; database B is the data of Shanghai; database C is the data of Guangzhou. If it is only configured according to the shard item, the developer needs to know that 0 means Beijing; 1 means Shanghai; 2 means Guangzhou. Reasonable use of personalized parameters can make the code more readable. If the configuration is 0=Beijing, 1=Shanghai, 2=Guangzhou, then the enumeration values of Beijing, Shanghai, and Guangzhou can be used directly in the code to complete the sharding items and business logic corresponding relationship.
AverageAllocationJobShardingStrategy
The sharding strategy based on the average distribution algorithm is also the default sharding strategy. If the shards are not divisible, the remaining shards are added to the servers with sequential numbers in turn:
1, 3 servers, divided into 9 pieces. The first server is [0,1,2], the second server is [3,4,5], and the third server is [6,7,8]; 2 and 3
servers are divided into 8 pieces. The first server is [0,1,6], the second server is [2,3,7], and the third server is [4,5];
3, 3 servers are divided into 10 pieces. The first station is [0,1,2,9], the second station is [3,4,5], and the third station is [6,7,8];
OdevitySortByNameJobShardingStrategy
The sharding strategy of the IP ascending and descending algorithm is determined according to the odd and even numbers of the hash value of the job name. Odd numbers are in ascending order of IPs, and even numbers are in descending order of IPs, which is used to evenly distribute loads to different servers for different jobs.
Custom sharding strategy
Implement the JobShardingStrategy interface
package com.feenix.elasticjob.strategy;
import org.apache.shardingsphere.elasticjob.infra.handler.sharding.JobInstance;
import org.apache.shardingsphere.elasticjob.infra.handler.sharding.JobShardingStrategy;
import java.util.*;
public class CustomJobShardingStrategy implements JobShardingStrategy {
@Override
public Map<JobInstance, List<Integer>> sharding(List<JobInstance> jobInstances, String jobName, int shardingTotalCount) {
// 作业分片加入容器
ArrayList<Integer> customShardingList = new ArrayList<>();
for (int i = 0; i < shardingTotalCount; i++) {
customShardingList.add(i);
}
// 将容器中的作业分片项顺序打乱
Collections.shuffle(customShardingList);
// 模拟AverageAllocationJobShardingStrategy算法
Map<JobInstance, List<Integer>> result = shardingCustom(jobInstances, shardingTotalCount, customShardingList);
addCustom(jobInstances, shardingTotalCount, result, customShardingList);
return result;
}
private Map<JobInstance, List<Integer>> shardingCustom(final List<JobInstance> shardingUnits,
final int shardingTotalCount,
final ArrayList<Integer> customShardingList) {
Map<JobInstance, List<Integer>> result = new LinkedHashMap<>(shardingUnits.size(), 1);
int itemCountPerSharding = shardingTotalCount / shardingUnits.size();
int count = 0;
for (JobInstance each : shardingUnits) {
// 每个作业服务器申请的作业分片项列表，容量是itemCountPersharding+1，为每个作业最大的分片项
List<Integer> shardingItems = new ArrayList<>(itemCountPerSharding + 1);
for (int i = count * itemCountPerSharding; i < (count + 1) * itemCountPerSharding; i++) {
shardingItems.add(customShardingList.get(i));
}
result.put(each, shardingItems);
count++;
}
return result;
}
private void addCustom(final List<JobInstance> shardingUnits,
final int shardingTotalCount,
final Map<JobInstance, List<Integer>> shardingResults,
final ArrayList<Integer> customShardingList) {
int aliquant = shardingTotalCount % shardingUnits.size();
int count = 0;
for (Map.Entry<JobInstance, List<Integer>> entry : shardingResults.entrySet()) {
if (count < aliquant) {
entry.getValue().add(customShardingList.get(shardingTotalCount / shardingUnits.size() * shardingUnits.size() + count));
}
count++;
}
}
@Override
public String getType() {
return "CUSTOM";
}
}
Create configuration file
In the resources.META-INF.services directory, create a file with the full path name of the interface of the custom policy implementation class, and the content is the full path of the custom policy implementation class
Use IDEA to open two different service instances
package com.feenix.elasticjob.service;
import com.feenix.elasticjob.service.impl.FeenixJobImpl;
import org.apache.shardingsphere.elasticjob.api.JobConfiguration;
import org.apache.shardingsphere.elasticjob.lite.api.bootstrap.impl.ScheduleJobBootstrap;
import org.apache.shardingsphere.elasticjob.reg.base.CoordinatorRegistryCenter;
import org.apache.shardingsphere.elasticjob.reg.zookeeper.ZookeeperConfiguration;
import org.apache.shardingsphere.elasticjob.reg.zookeeper.ZookeeperRegistryCenter;
public class Application {
public static void main(String[] args) {
new ScheduleJobBootstrap(createRegistryCenter(), new FeenixJobImpl(), createJobConfiguration()).schedule();
}
// 构建Zookeeper注册中心
private static CoordinatorRegistryCenter createRegistryCenter() {
ZookeeperConfiguration zkConfiguration = new ZookeeperConfiguration("192.168.0.31:2181",
"feenix-job");
zkConfiguration.setConnectionTimeoutMilliseconds(100000);
zkConfiguration.setMaxRetries(10);
ZookeeperRegistryCenter zkRegistryCenter = new ZookeeperRegistryCenter(zkConfiguration);
zkRegistryCenter.init();
return zkRegistryCenter;
}
// 作业配置
private static JobConfiguration createJobConfiguration() {
String jobs = "0=宋远桥,1=俞莲舟,2=俞岱岩,3=张松溪,4=张翠山,5=殷梨亭,6=莫声谷";
return JobConfiguration.newBuilder("FeenixJobImpl", 7)
.cron("0/3 * * * * ?")
.shardingItemParameters(jobs)
// 使用自定义的作业分片策略
.jobShardingStrategyType("CUSTOM")
// 允许客户端配置覆盖注册中心
.overwrite(true)
// 故障转移
.failover(true)
.build();
}
}
The value of jobShardingStrategyType should be set to "CUSTOM", because when customizing the sharding strategy, the custom strategy type has been written as "CUSTOM".
It can be seen that when two instances are started at the same time, the job is assigned to two different instances for execution according to the customized policy
