One, elastic-job framework
Developed based on the Quartz framework and zooepper.
It mainly solves the problem that jobs under the cluster will be executed multiple times, and the cluster cannot scale jobs horizontally (execution efficiency problem). Features are as follows:
- Distributed scheduling coordination (provided by zookeeper)
- Flexible expansion and contraction
- Failover (when a node fails, other nodes will take over the job execution)
- Missed job re-triggering
- Consistency of job shards to ensure that the same shard is executed in only one instance in a distributed environment (no multiple executions will occur)
- Rich job types (simple, dataflow)
- Spring integration and namespace provision (SpringJobScheduler)
- Provide operation and maintenance platform (elastic-job-lite-console)
2. How to use elastic-job?
Need to start zookeeper first,
1. Introduce dependencies
<dependency>
<groupId>com.dangdang</groupId>
<artifactId>elastic-job-lite-spring</artifactId>
<version>2.1.5</version>
</dependency>2. Two ways to create job classes:
2.1 Create a job class to implement the SimpleJob interface (simple job task)
Features:
public class MySimpleJob implements SimpleJob {
@Override
public void execute(ShardingContext shardingContext) {
int shardingItem = shardingContext.getShardingItem();//当前分片项
String jobParameter = shardingContext.getJobParameter();//当前分片自定义的参数，例如按照类型分片（txt、jpg、pdf等）
//这里可以根据分片的信息，去数据查询对应的数据，进行处理
//例如按照id奇偶数去分片。或按照类型等字段分片处理数据，提高性能
System.out.println(new Date()+":"+JSON.toJSONString(shardingContext));
}
}2.2 or implement the DataFlowJob interface (data flow job task)
Features:
public class MyDateFlowJob implements DataflowJob<Object> {
@Override
public List<Object> fetchData(ShardingContext shardingContext) {
//当返回有数据时，则执行下边的processData方法，一直到没有数据返回（执行完了），本次作业完毕
//等待进入下一次作业
return null;
}
@Override
public void processData(ShardingContext shardingContext, List<Object> data) {
System.out.println(new Date()+"处理数据");
}
}3. Create a configuration class
Every step is clearly written in the code.
@Configuration
public class ElasticJobConfig {
@Bean
public SpringJobScheduler dataFlowJobScheduler(DataSource dataSource){
//1、创建zookeeper配置（zookeeper地址，命名空间）
ZookeeperConfiguration zookeeperConfiguration = new ZookeeperConfiguration("localhost:2181", "elastic-job");
//2、创建zookeeper配置中心,并初始化（根据zookeeper配置创建）
ZookeeperRegistryCenter zookeeperRegistryCenter = new ZookeeperRegistryCenter(zookeeperConfiguration);
zookeeperRegistryCenter.init();
//3、创建作业配置（作业名称、cron表达式、分片数量）
JobCoreConfiguration jobCoreConfiguration = JobCoreConfiguration.newBuilder("myDataflowJob", "0/30 * * * * ?", 5)
.jobParameter("this jobParameter")//作业参数
.description("这个作业是干什么的？描述一下")//作业描述
.failover(true)//开始失效转移。当某节点挂掉之后，作业会分配到其他节点继续完成
.shardingItemParameters("0=txt,1=jpg,2=pdf,3=word,4=excel")//分片附带的参数
.build();
//4、创建数据流作业配置（根据作业配置、实现了DataflowJob实现类的全限定名、是否流式处理创建）
DataflowJobConfiguration dataflowJobConfiguration = new DataflowJobConfiguration(jobCoreConfiguration, MyDateFlowJob.class.getCanonicalName(),true);
//5、创建精简版作业配置（根据简单作业配置创建）
LiteJobConfiguration liteJobConfiguration = LiteJobConfiguration.newBuilder(dataflowJobConfiguration)
.jobShardingStrategyClass("com.dangdang.ddframe.job.lite.api.strategy.impl.RotateServerByNameJobShardingStrategy")//配置作业分片策略类
//默认elastic-job提供了三种分片策略：
//1、AverageAllocationJobShardingStrategy 平均分配策略（默认）
//2、OdevitySortByNameJobShardingStrategy 根据作业名hash值奇偶数升降ip分配策略
//3、RotateServerByNameJobShardingStrategy 根据作业名hash值对服务器列表轮转分配策略
//你还可以自定义分配策略实现JobShardingStrategy接口
.overwrite(true)//覆盖注册中心配置。如果不覆盖，一次注册后，配置便再也不能改变了。
.monitorPort(8889)//作业辅助监控端口。当生产环境不能连接作业监控时，可通过此端口导出日志查看。
.build();
//6、创建作业事件配置（可选）（根据数据源创建）
//作用：把作业的执行过程写入数据库表中
JobEventConfiguration jobEventConfiguration = new JobEventRdbConfiguration(dataSource);
//7、创建Elastic监听器（可选）
//作用：可以在作业执行的前后做某些操作
MyDistributeOnceElasticJobListener listener = new MyDistributeOnceElasticJobListener(0,0);
//8、创建spring作业定时器,并初始化（根据作业类的实例、zookeeper注册中心、精简版作业配置、[作业事件配置]、[elastic监听器]）
//作用:正式创建作业
SpringJobScheduler springJobScheduler = new SpringJobScheduler(new MyDateFlowJob(), zookeeperRegistryCenter, liteJobConfiguration, jobEventConfiguration, listener);
springJobScheduler.init();
return springJobScheduler;
}
}4. Configure the elastic-job listener (the method in the listener will be called every time the job is executed) (optional)
public class MyDistributeOnceElasticJobListener extends AbstractDistributeOnceElasticJobListener {
public MyDistributeOnceElasticJobListener(long startedTimeoutMilliseconds, long completedTimeoutMilliseconds) {
super(startedTimeoutMilliseconds, completedTimeoutMilliseconds);
}
@Override
public void doBeforeJobExecutedAtLastStarted(ShardingContexts shardingContexts) {
System.out.println("监听器--执行之前执行的方法");
}
@Override
public void doAfterJobExecutedAtLastCompleted(ShardingContexts shardingContexts) {
System.out.println("监听器--执行之后执行的方法");
}
}
3. How to use the elastic-job visualization console?
1. Download the project: https://github.com/apache/shardingsphere-elasticjob/tree/2.1.5/elastic-job-lite , the download is complete and decompressed
2. Maven package packages the elastic-job-lite-console project, and then there is an elastic-job-lite-console-2.1.5.tar.gz file in the target directory
3. Unzip the elastic-job-lite-console-2.1.5.tar.gz file, enter the bin directory and execute start.sh to start
4. Visit http://localhost:8899/ . Account password: root/root
Enter the console
1. Global configuration, add zookeeper information and mysql information
2. Job operation: you can see job information and server information
3. Job history: you can view the execution of the job
4. How to use the operation auxiliary port 8889?
When zookeeper cannot be used in the production environment, we can export the job execution status through the job auxiliary port to facilitate debugging.
1. The job auxiliary port needs to be opened in the LiteJobConfiguration configuration.
2. Need to use dump command and nc command, no need to install
3. Execute the echo dump|nc 127.0.0.1 9888> job_debug_dump.txt command