Distributed scheduling Elastic-job

Distributed scheduling Elastic-job

1 Overview

---

1.1 What is task scheduling?

We can think about solutions for the following business scenarios:

• An e-commerce platform needs to issue a batch of coupons every day at 10 am, 3 pm and 8 pm
• A certain bank system needs to send SMS reminders three days before the due date of credit card repayment.
• A financial system needs to settle the financial data of the previous day at 0:10 am every day, statistical summary

The above scenario is the problem that task scheduling needs to solve

Task scheduling is the process of automatically completing specific tasks and performing tasks at a specific time agreed

We often use the scheduled task annotation @Scheduled provided in Spring to affix this annotation to the method in the business class

@Scheduled(cron = "0/20 * * * * ? ")
public void doWork(){
    
    
//doSomething 
}

Then paste the @EnableScheduling annotation on the startup class

1.2 Why distributed scheduling is needed

I feel that the annotation provided by Spring can complete the task scheduling function. It seems to have solved the problem perfectly. Why is it still needed to be distributed?

The main reasons are as follows:

1. Single-machine processing limit : Originally, 10,000 orders needed to be processed in 1 minute, but now it needs to process 100,000 orders in 1 minute; it used to take 1 hour for a statistics, but now it takes 10 minutes for the business side to calculate it. You might say that you can also do multi-threading and single-machine multi-process processing. Indeed, multi-threaded parallel processing can improve the processing efficiency per unit time, but the capabilities of a single machine are limited after all (mainly CPU, memory and disk), and there will always be situations that a single machine cannot handle.

2. High availability : The stand-alone version of fixed task scheduling can only run on one machine. If an abnormality occurs in the program or system, the function will be unavailable. Although it can be implemented stably in a stand-alone program, there is always a chance of encountering failures that are not caused by the program, and this is unacceptable for the core functions of a system.

3. Prevent repeated execution : In stand-alone mode, there is no problem with scheduled tasks. But when we deploy multiple services and each service has scheduled tasks, if there is no reasonable control at the same time, only one scheduled task will start executing. At this time, the result of scheduled execution will be There may be confusion and errors

At this time, distributed task scheduling is needed to achieve.

1.3 Elastic-Job introduction

Elastic-Job is a distributed scheduling solution, open sourced by Dangdang. It consists of two independent sub-projects, Elastic-job-Lite and Elastic-Job-Cloud. Using Elastic-Job can quickly Implement distributed task scheduling.

Elastic-Job address: https://shardingsphere.apache.org/elasticjob/

function list:

• Distributed scheduling coordination

  • In a distributed environment, tasks can be executed according to the specified scheduling policy and can avoid repeated execution of multiple instances of the same task.

• Rich scheduling strategies

  • Execute scheduled tasks based on the mature scheduled task job framework Quartz cron expression.

• Flexible expansion and contraction

  • When an instance is added to the cluster, it should be able to be elected to perform tasks; when an instance is removed from the cluster, the tasks it performs can be transferred to other instances for execution.

• failover

  • After an instance fails to execute a task, it will be transferred to another instance for execution.

• Missed task execution retriggers

  • If a job misses execution for some reason, the erroneously executed job will be automatically recorded and triggered automatically after the next job is completed.

• Support parallel scheduling

  • Supports task sharding. Task sharding refers to dividing a task into multiple small tasks and executing them simultaneously on multiple instances.

• Job Sharding Consistency

  • When a task is divided into pieces, it is guaranteed that there is only one execution instance of the same piece in the distributed environment.

• Support job lifecycle operations

• Tasks can be started and stopped dynamically.

• Rich job types

  • Supports three job types: Simple, DataFlow, and Script

    

  System architecture diagram

  

2.Elastic-Job quick start

---

2.1 Environment Construction

2.1.1 Version 02. Requirements

• JDK requires version 1.7 or above

• Maven requires version 3.0.4 and above

• Zookeeper requires version 3.4.6 or above

2.1.2 Zookeeper installation and operation

1. 解压zookeeper-3.4.11.tar.gz, 进入conf目录, 复制zoo_sample.cfg文件, 命名为:zoo.cfg
2. 进入bin目录, 运行zkServer.cmd就可以了.
3. 解压ZooInspector.zip, 运行jar文件

zookeeper client visualization tool

2.1.3 Create Maven project

Add the following dependencies

<dependency>
	<groupId>com.dangdang</groupId>
 	<artifactId>elastic-job-lite-core</artifactId>
 	<version>2.1.5</version>
</dependency>

2.2 Code implementation

2.2.1 Task class

package com.xiaoge;

import com.dangdang.ddframe.job.api.ShardingContext;
import com.dangdang.ddframe.job.api.simple.SimpleJob;

import java.util.Date;

public class MyElasticJob implements SimpleJob {
    
    
    public void execute(ShardingContext shardingContext) {
    
    
        System.out.println("定时任务开始====>" + new Date());
    }
}

2.2.2 Configuration class

package com.xiaoge;

import com.dangdang.ddframe.job.config.JobCoreConfiguration;
import com.dangdang.ddframe.job.config.simple.SimpleJobConfiguration;
import com.dangdang.ddframe.job.lite.api.JobScheduler;
import com.dangdang.ddframe.job.lite.config.LiteJobConfiguration;
import com.dangdang.ddframe.job.reg.base.CoordinatorRegistryCenter;
import com.dangdang.ddframe.job.reg.zookeeper.ZookeeperConfiguration;
import com.dangdang.ddframe.job.reg.zookeeper.ZookeeperRegistryCenter;

public class JobDemo {
    
    
    public static void main(String[] args) {
    
    
        // JobScheduler(注册中心对象， 任务配置对象)
        new JobScheduler(createRegistryCenter(), createJobConfiguration()).init();
    }

    // 注册中心
    private static CoordinatorRegistryCenter createRegistryCenter() {
    
    
        // 配置zk地址,调度任务的组名
        ZookeeperConfiguration zookeeperConfiguration = new ZookeeperConfiguration("localhost:2181", "elastic-job-demo");
        // 设置节点超时时间
        zookeeperConfiguration.setSessionTimeoutMilliseconds(100);
        // ZookeeperRegistryCenter("zookeeper地址", "项目名")
        CoordinatorRegistryCenter regCenter = new ZookeeperRegistryCenter(zookeeperConfiguration);
        regCenter.init();
        return regCenter;
    }

    // 定时任务配置
    private static LiteJobConfiguration createJobConfiguration() {
    
    
        // 定义作业核⼼配置 newBuilder("任务名称", "cron表达式", "分片数量")
        JobCoreConfiguration simpleCoreConfig =
                JobCoreConfiguration.newBuilder("myElasticJob", "0/10 * * * * ?", 1).build();
        // 定义SIMPLE类型配置 MyElasticJob.class.getCanonicalName()--->获取这个类的权限定类名
        SimpleJobConfiguration simpleJobConfig = new SimpleJobConfiguration(simpleCoreConfig, MyElasticJob.class.getCanonicalName());
        // 定义Lite作业根配置 (overwrite(true) 表示zookeeper里面的配置可以覆盖, 如果为false, 设置了一次cron表达式, 第二次修改表达式是不生效的)
        LiteJobConfiguration simpleJobRootConfig = LiteJobConfiguration.newBuilder(simpleJobConfig).overwrite(true).build();
        return simpleJobRootConfig;
    }
}

2.2.3 Testing

• Run a single program to see whether tasks are scheduled according to the contents of the cron expression.

• Run multiple programs to see if there will only be one instance for task scheduling.

• After running multiple programs, shut down the process that is scheduling tasks and check whether other processes can continue to schedule tasks.

3.SpringBoot integrates Elastic-Job

---

3.1 Add Maven dependencies

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.xiaoge</groupId>
    <artifactId>elastic-job-boot</artifactId>
    <version>1.0-SNAPSHOT</version>

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.1.3.RELEASE</version>
    </parent>

    <properties>
        <maven.compiler.source>8</maven.compiler.source>
        <maven.compiler.target>8</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>

        <dependency>
            <groupId>com.dangdang</groupId>
            <artifactId>elastic-job-lite-spring</artifactId>
            <version>2.1.5</version>
        </dependency>

        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
        </dependency>
    </dependencies>

</project>

3.2 Related configurations

Because the address of the configuration center is not fixed, we should configure this address information in the configuration file, so add the following configuration in the configuration file application.yml:

elasticjob:
  url: localhost:2181
  group-name: elastic-job-boot

zk registration center configuration class:

@Bean
public CoordinatorRegistryCenter registryCenter(@Value("${elasticjob.url}") String zookeeperUrl, @Value("${elasticjob.group-name}") String groupName) {
    
    
    // 配置zk地址,调度任务的组名
    ZookeeperConfiguration zookeeperConfiguration = new ZookeeperConfiguration(zookeeperUrl, groupName);
    // 设置节点超时时间
    zookeeperConfiguration.setSessionTimeoutMilliseconds(100);
    // ZookeeperRegistryCenter("zookeeper地址", "项目名")
    CoordinatorRegistryCenter regCenter = new ZookeeperRegistryCenter(zookeeperConfiguration);
    regCenter.init();
    return regCenter;
}

Task scheduling configuration class:

/**
     * todo 注意这个方法不能交给 spring 管理, 你要让它是个公共的方法,
     *      传递不同的jobName(任务名称), cron(cron表达式), shardingTotalCount(分片数量) 生成不同的LiteJobConfiguration, 因为环境不同任务配置不同.
     *      也有可能别的任务需要这个方法创建
     * @return
     */
public LiteJobConfiguration createJobConfiguration(Class<?> clazz, String cron, Integer shardingTotalCount, String shardingParam) {
    
    
    // 定义作业核⼼配置 newBuilder("任务名称", "cron表达式", "分片数量")
    JobCoreConfiguration.Builder jobBuilder = JobCoreConfiguration.newBuilder(clazz.getSimpleName(), cron, shardingTotalCount);
    if (!StringUtils.isEmpty(shardingParam)) {
    
    
        // 分片参数
        jobBuilder = jobBuilder.shardingItemParameters(shardingParam);
    }
    // SimpleJob配置
    // 定义SIMPLE类型配置 MyElasticJob.class.getCanonicalName()--->获取这个类的权限定类名
    SimpleJobConfiguration simpleJobConfiguration = new SimpleJobConfiguration(jobBuilder.build(), clazz.getCanonicalName());
    

    // 定义Lite作业根配置 (overwrite(true) 表示zookeeper里面的配置可以覆盖, 如果为false, 设置了一次cron表达式, 第二次修改表达式是不生效的)
    LiteJobConfiguration simpleJobRootConfig = LiteJobConfiguration.newBuilder(simpleJobConfiguration).overwrite(true).build();
    return simpleJobRootConfig;
}

4. Case requirements

---

Requirements: There are some columns of data in the database, and these data need to be backed up. After the backup is completed, the status of the data is modified and marked as backed up.

4.1 Initialization data

Import elastic-job-demo.sql data into the database

4.2 Integrate Druid&MyBatis

4.2.1 Add dependencies

<dependency>
    <groupId>com.alibaba</groupId>
    <artifactId>druid</artifactId>
    <version>1.1.10</version>
</dependency>
<dependency>
    <groupId>org.mybatis.spring.boot</groupId>
    <artifactId>mybatis-spring-boot-starter</artifactId>
    <version>1.2.0</version>
</dependency>
<!--mysql驱动-->
<dependency>
    <groupId>mysql</groupId>
    <artifactId>mysql-connector-java</artifactId>
</dependency>

4.2.2 Add configuration

spring:
  datasource:
    url: jdbc:mysql://localhost:3306/elastic-job-demo?serverTimezone=GMT%2B8
    driverClassName: com.mysql.jdbc.Driver
    type: com.alibaba.druid.pool.DruidDataSource
    username: root
    password: root

4.2.3 Add entity class

package com.xiaoge.domain;

import lombok.Data;

@Data
public class FileCustom {
    
    
    //唯⼀标识
    private Long id;
    //⽂件名
    private String name;
    //⽂件类型
    private String type;
    //⽂件内容
    private String content;
    //是否已备份
    private Boolean backedUp = false;

    public FileCustom() {
    
    
    }

    public FileCustom(Long id, String name, String type, String content) {
    
    
        this.id = id;
        this.name = name;
        this.type = type;
        this.content = content;
    }
}

4.2.4 Add Mapper processing class

package com.xiaoge.mapper;

import com.xiaoge.domain.FileCustom;
import org.apache.ibatis.annotations.Mapper;
import org.apache.ibatis.annotations.Param;
import org.apache.ibatis.annotations.Select;
import org.apache.ibatis.annotations.Update;

import java.util.List;

@Mapper
public interface FileCustomMapper {
    
    
    @Select("select * from t_file_custom where backedUp = 0")
    List<FileCustom> selectAll();

    @Update("update t_file_custom set backedUp = #{state} where id = #{id}")
    int changeState(@Param("id") Long id, @Param("state") int state);
}

4.3 Realization of business functions

4.3.1 Add task class

package com.xiaoge.service;

import com.dangdang.ddframe.job.api.ShardingContext;
import com.dangdang.ddframe.job.api.simple.SimpleJob;
import com.xiaoge.domain.FileCustom;
import com.xiaoge.mapper.FileCustomMapper;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;

import java.util.List;
import java.util.concurrent.TimeUnit;

@Slf4j
@Component
public class FileCustomElasticJob implements SimpleJob {
    
    
    @Autowired
    private FileCustomMapper fileCustomMapper;

    @Override
    public void execute(ShardingContext shardingContext) {
    
    
        doWork();
    }
    private void doWork(){
    
    
        List<FileCustom> fileList = fileCustomMapper.selectAll();
        System.out.println("需要备份⽂件个数:"+fileList.size());
        for(FileCustom fileCustom:fileList){
    
    
            backUpFile(fileCustom);
        }
    }
    private void backUpFile(FileCustom fileCustom){
    
    
        try {
    
    
            //模拟备份动作
            TimeUnit.SECONDS.sleep(1);
        } catch (InterruptedException e) {
    
    
            e.printStackTrace();
        }
        System.out.println("执⾏⽂件备份====>"+fileCustom);
        fileCustomMapper.changeState(fileCustom.getId(),1);
    }
}

4.3.2 Add task scheduling configuration

Add this Bean in the configuration class

/**
     * todo 注意一个ElasticJob里面不管有多少实例, 只会有一个被调度, 那就是zookeeper选出来的leader
     * @param myElasticJob
     * @param regCenter
     * @return
     */
@Bean(initMethod = "init")
public SpringJobScheduler initSpringScheduler(ElasticJob myElasticJob, CoordinatorRegistryCenter regCenter) {
    
    
    LiteJobConfiguration simpleJobRootConfig = createJobConfiguration(myElasticJob.getClass(), "0/10 * * * * ?", 1);
    return new SpringJobScheduler(myElasticJob, regCenter, simpleJobRootConfig);
}

4.4 Tests & Questions

For high availability, we will perform cluster operations on this project to ensure that if one machine hangs up, the other one can continue to work. However, in the case of a cluster, the scheduling task will only run on one machine. If a single machine Task scheduling is time-consuming and consumes resources. The consumption of this machine is still relatively large, but at this time, other machines are idle. How to make reasonable use of other machines in the cluster and how to make tasks execute smoothly? Faster? At this time, Elastic-Job provides the function of task scheduling fragmentation.

5. Sharding concept

---

Job sharding refers to the distributed execution of tasks. A task needs to be split into multiple independent task items, and then one or several distributed items are executed by distributed application instances.

For example: in the case of file backup in the Elastic-Job Quick Start, there are two servers, each running an application instance. In order to execute the job quickly, the task can be divided into 4 slices, and each application instance executes two slices. The job traversal data logic should be: instance 1 searches for text and image type files to perform backup, and instance 2 searches for radio and vedio type files to perform backup. If the number of application instances increases to 4 due to server capacity expansion, the logic of job traversal data should be: 4 instances process files of text, image, radio, and video types respectively.

For example: in the case of file backup in the Elastic-Job Quick Start, there are two servers, each running an application instance. In order to execute the job quickly, the task can be divided into 4 slices, and each application instance executes two slices. The job traversal data logic should be: instance 1 searches for text and image type files to perform backup, and instance 2 searches for radio and vedio type files to perform backup. If the number of application instances increases to 4 due to server capacity expansion, the logic of job traversal data should be: 4 instances process files of text, image, radio, and video types respectively.

Decoupling sharding items from business processing

Elastic-Job does not directly provide data processing functions. The framework will only allocate shard items to each running job server. Developers need to handle the correspondence between shard items and real data by themselves.

maximize resource utilization

Set the shard items to be larger than the server's data, preferably a multiple of the server. The job will make reasonable use of distributed resources and dynamically allocate shard items.

For example: 3 servers, divided into 10 shards, then the sharding item results are server A=0,1,2; server B=3,4,5; server C=6,7,8,9. If server C crashes , then the shard item allocation results are server A=0,1,2,3,4; server B=5,6,7,8,9. Maximize the use of existing shard items without losing the shard items. resources to improve throughput.

6. Transform the case into task sharding

---

6.1 Configuration class modification

Add the number of shards and shard parameters in the task configuration class.

@Bean(initMethod = "init")
public SpringJobScheduler initFileCustomElasticJob(FileCustomElasticJob
                                                   fileCustomElasticJob){
    
    
    SpringJobScheduler springJobScheduler = new SpringJobScheduler(
        fileCustomElasticJob,
        registryCenter,
        createJobConfiguration(FileCustomElasticJob.class,"0 0/1 * * *
                               ?",4,"0=text,1=image,2=radio,3=vedio"));
    return springJobScheduler;
}

6.2 Added job sharding logic

package com.xiaoge.service;

import com.dangdang.ddframe.job.api.ShardingContext;
import com.dangdang.ddframe.job.api.simple.SimpleJob;
import com.xiaoge.domain.FileCustom;
import com.xiaoge.mapper.FileCustomMapper;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;

import java.util.List;
import java.util.concurrent.TimeUnit;

@Slf4j
@Component
public class FileCustomElasticJob implements SimpleJob {
    
    
    @Autowired
    private FileCustomMapper fileCustomMapper;

    @Override
    public void execute(ShardingContext shardingContext) {
    
    
        long threadId = Thread.currentThread().getId();
        log.info("线程ID: {}, 任务的名称: {}, 任务参数: {}, 分片个数: {}, 分片索引号: {}, 分片参数: {}",
                threadId,
                shardingContext.getJobName(),
                shardingContext.getJobParameter(),
                shardingContext.getShardingTotalCount(),
                shardingContext.getShardingItem(),
                shardingContext.getShardingParameter()
        );
        doWork(shardingContext.getShardingParameter());
    }

    private void doWork(String shardingParameter) {
    
    
        List<FileCustom> fileList = fileCustomMapper.selectFileCustomByType(shardingParameter);
        log.info("需要备份⽂件个数{}: {}", shardingParameter, fileList.size());
        for (FileCustom fileCustom : fileList) {
    
    
            backUpFile(fileCustom);
        }
    }

    private void backUpFile(FileCustom fileCustom) {
    
    
        try {
    
    
            //模拟备份动作
            TimeUnit.SECONDS.sleep(1);
        } catch (InterruptedException e) {
    
    
            e.printStackTrace();
        }
        System.out.println("执⾏⽂件备份====>" + fileCustom);
        fileCustomMapper.changeState(fileCustom.getId(), 1);
    }
}

6.3 Mapper class modification

package com.xiaoge.mapper;

import com.xiaoge.domain.FileCustom;
import org.apache.ibatis.annotations.Mapper;
import org.apache.ibatis.annotations.Param;
import org.apache.ibatis.annotations.Select;
import org.apache.ibatis.annotations.Update;

import java.util.List;

@Mapper
public interface FileCustomMapper {
    
    
    @Select("select * from t_file_custom where backedUp = 0")
    List<FileCustom> selectAll();

    @Select("select * from t_file_custom where backedUp = 0 and type = #{type}")
    List<FileCustom> selectFileCustomByType(@Param("type") String type);

    @Update("update t_file_custom set backedUp = #{state} where id = #{id}")
    int changeState(@Param("id") Long id, @Param("state") int state);
}

6.4 Testing

• How is task sharding performed when there is only one machine?

• How is task sharding performed when there are multiple machines?

7.Dataflow type scheduling tasks

---

Scheduled tasks of the Dataflow type need to implement the Dataflowjob interface. This interface provides two methods for coverage, which are used to fetch (fetchData) and process (processData) data. We will continue to modify the example.

Scheduled tasks of the Dataflow type need to implement the Dataflowjob interface. This interface provides two methods for coverage, which are used to fetch (fetchData) and process (processData) data. We will continue to modify the example.

7.1 Task class

package com.xiaoge.service;

import com.dangdang.ddframe.job.api.ShardingContext;
import com.dangdang.ddframe.job.api.dataflow.DataflowJob;
import com.xiaoge.domain.FileCustom;
import com.xiaoge.mapper.FileCustomMapper;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Configuration;
import org.springframework.stereotype.Component;

import java.util.List;
import java.util.concurrent.TimeUnit;

/**
 * TODO 处理大数据量的时候用那个DataFlow这种方式
 *
 * @author <a href="mailto:[email protected]">Zhang Xiao</a>
 * @since
 */
@Component
public class FileDataFlowJob implements DataflowJob<FileCustom> {
    
    

    @Autowired
    private FileCustomMapper fileCustomMapper;

    // 抓取数据
    @Override
    public List<FileCustom> fetchData(ShardingContext shardingContext) {
    
    
        System.out.println("开始抓取数据...........");
        return fileCustomMapper.selectLimit(shardingContext.getShardingParameter(), 2);
    }

    // 处理数据
    @Override
    public void processData(ShardingContext shardingContext, List<FileCustom> fileCustomList) {
    
    
        fileCustomList.forEach(fileCustom -> {
    
    
            backUpFile(fileCustom);
        });
    }

    private void backUpFile(FileCustom fileCustom) {
    
    
        System.out.println("备份的方法名: " + fileCustom.getName() + "备份的类型: " + fileCustom.getType());
        try {
    
    
            //模拟备份动作
            TimeUnit.SECONDS.sleep(1);
        } catch (InterruptedException e) {
    
    
            e.printStackTrace();
        }
        System.out.println("执⾏⽂件备份====>" + fileCustom);
        fileCustomMapper.changeState(fileCustom.getId(), 1);
    }
}

7.2 Configuration class

package com.xiaoge.config;

import com.dangdang.ddframe.job.config.JobCoreConfiguration;
import com.dangdang.ddframe.job.config.JobTypeConfiguration;
import com.dangdang.ddframe.job.config.dataflow.DataflowJobConfiguration;
import com.dangdang.ddframe.job.config.simple.SimpleJobConfiguration;
import com.dangdang.ddframe.job.event.JobEventConfiguration;
import com.dangdang.ddframe.job.event.rdb.JobEventRdbConfiguration;
import com.dangdang.ddframe.job.lite.config.LiteJobConfiguration;
import com.dangdang.ddframe.job.lite.spring.api.SpringJobScheduler;
import com.dangdang.ddframe.job.reg.base.CoordinatorRegistryCenter;
import com.dangdang.ddframe.job.reg.zookeeper.ZookeeperConfiguration;
import com.dangdang.ddframe.job.reg.zookeeper.ZookeeperRegistryCenter;
import com.xiaoge.service.FileCustomElasticJob;
import com.xiaoge.service.FileDataFlowJob;
import org.apache.commons.lang3.StringUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import javax.sql.DataSource;

/**
 * TODO
 *
 * @author <a href="mailto:[email protected]">Zhang Xiao</a>
 * @since
 */
@Configuration
public class ElasticJobConfig {
    
    

    /**
     * todo 注意一个ElasticJob里面不管有多少实例, 只会有一个被调度, 那就是zookeeper选出来的leader
     * @param myElasticJob
     * @param regCenter
     * @return
     */
//    @Bean(initMethod = "init")
//    public SpringJobScheduler testScheduler(ElasticJob myElasticJob, CoordinatorRegistryCenter regCenter) {
    
    
//        LiteJobConfiguration simpleJobRootConfig = createJobConfiguration(myElasticJob.getClass(), "0/10 * * * * ?", 1);
//        return new SpringJobScheduler(myElasticJob, regCenter, simpleJobRootConfig);
//    }

//    @Bean(initMethod = "init")
//    public SpringJobScheduler fileScheduler(FileCustomElasticJob fileCustomElasticJob, CoordinatorRegistryCenter regCenter){
    
    
//        SpringJobScheduler springJobScheduler = new SpringJobScheduler(fileCustomElasticJob,regCenter,createJobConfiguration(fileCustomElasticJob.getClass(),"0 0/1 * * * ?",4, "0=text,1=image,2=radio,3=vedio", false));
//        return springJobScheduler;
//    }

    @Bean(initMethod = "init")
    public SpringJobScheduler fileDataFlowScheduler(FileDataFlowJob fileDataFlowJob, CoordinatorRegistryCenter regCenter){
    
    
        SpringJobScheduler springJobScheduler = new SpringJobScheduler(fileDataFlowJob,regCenter,createJobConfiguration(fileDataFlowJob.getClass(),"0 0/1 * * * ?",4, "0=text,1=image,2=radio,3=vedio", true));
        return springJobScheduler;
    }

//    @Bean(initMethod = "init")
//    public SpringJobScheduler test1Scheduler(ElasticJob myElasticJob1, CoordinDataRevisionatorRegistryCenter regCenter) {
    
    
//        LiteJobConfiguration simpleJobRootConfig = createJobConfiguration(myElasticJob1.getClass(), "0/3 * * * * ?", 1);
//        return new SpringJobScheduler(myElasticJob1, regCenter, simpleJobRootConfig);
//    }

    @Bean
    public CoordinatorRegistryCenter registryCenter(@Value("${elasticjob.url}") String zookeeperUrl, @Value("${elasticjob.group-name}") String groupName) {
    
    
        // 配置zk地址,调度任务的组名
        ZookeeperConfiguration zookeeperConfiguration = new ZookeeperConfiguration(zookeeperUrl, groupName);
        // 设置节点超时时间
        zookeeperConfiguration.setSessionTimeoutMilliseconds(100);
        // ZookeeperRegistryCenter("zookeeper地址", "项目名")
        CoordinatorRegistryCenter regCenter = new ZookeeperRegistryCenter(zookeeperConfiguration);
        regCenter.init();
        return regCenter;
    }

    /**
     * todo 注意这个方法不能交给 spring 管理, 你要让它是个公共的方法,
     *      传递不同的jobName(任务名称), cron(cron表达式), shardingTotalCount(分片数量) 生成不同的LiteJobConfiguration, 因为环境不同任务配置不同.
     *      也有可能别的任务需要这个方法创建
     * @return
     */
    public LiteJobConfiguration createJobConfiguration(Class<?> clazz, String cron, Integer shardingTotalCount, String shardingParam, boolean isDataFlow) {
    
    
        // 定义作业核⼼配置 newBuilder("任务名称", "cron表达式", "分片数量")
        JobCoreConfiguration.Builder jobBuilder = JobCoreConfiguration.newBuilder(clazz.getSimpleName(), cron, shardingTotalCount);
        if (!StringUtils.isEmpty(shardingParam)) {
    
    
            // 分片参数
            jobBuilder = jobBuilder.shardingItemParameters(shardingParam);
        }
        JobTypeConfiguration jobConfiguration;
        if (isDataFlow) {
    
    
            // DataflowJob配置
            jobConfiguration = new DataflowJobConfiguration(jobBuilder.build(), clazz.getCanonicalName(), true);
        } else {
    
    
            // SimpleJob配置
            // 定义SIMPLE类型配置 MyElasticJob.class.getCanonicalName()--->获取这个类的权限定类名
            jobConfiguration = new SimpleJobConfiguration(jobBuilder.build(), clazz.getCanonicalName());
        }

        // 定义Lite作业根配置 (overwrite(true) 表示zookeeper里面的配置可以覆盖, 如果为false, 设置了一次cron表达式, 第二次修改表达式是不生效的)
        LiteJobConfiguration simpleJobRootConfig = LiteJobConfiguration.newBuilder(jobConfiguration).overwrite(true).build();
        return simpleJobRootConfig;
    }



}

7.3 Testing

8. Operation and maintenance management

---

8.1 Event tracking

Elastic-Job-Lite provides JobEventConfiguration in the configuration, supports database configuration, and automatically creates two tables, JOB_EXECUTION_LOG and JOB_STATUS_TRACE_LOG, as well as various indexes in the database to obtain information about shortcut jobs.

8.1.1 Modify the Elastic-Job configuration class

Inject DataSource into the ElasticJobConfig configuration class

@Configuration
public class ElasticJobConfig {
    
    
 @Autowired
 private DataSource dataSource;
 ......
}

Add event tracking configuration in task configuration

@Bean(initMethod = "init")
    public SpringJobScheduler fileDataFlowScheduler(FileDataFlowJob fileDataFlowJob, CoordinatorRegistryCenter regCenter){
    
    
        // 日志监控, 它会自动在数据库生成两张表job_execution_log/job_status_trace_log
        // 配置会在任务执行的时间将任务执行的情况存储到数据源中
        JobEventConfiguration jobEventConfiguration = new JobEventRdbConfiguration(dataSource);
        SpringJobScheduler springJobScheduler = new SpringJobScheduler(fileDataFlowJob,regCenter,createJobConfiguration(fileDataFlowJob.getClass(),"0 0/1 * * * ?",4, "0=text,1=image,2=radio,3=vedio", true), jobEventConfiguration);
        return springJobScheduler;
}

8.1.2 Log information table

After startup, you will find that the following two tables have been added to the elastic-job-demo database.

job_execution_log

Recording the execution history of each job is divided into two steps:

1. Insert data into the database when the job starts executing.

2. When the job completes execution, update data to the database, update is_success, complete_time and failure_cause (if the task execution fails)

job_status_trace_log

Record the job status change trace table, and you can query the life track and running track of the job status change through the task_id of each job run.

8.2 Operation and maintenance console

An elastic-job-lite-console console is provided in elastic-job

design concept

1. This console is not directly related to Elastic-Job. It displays the job status by reading the registration center data of Elastic-Job, or updating the registration center data to modify the global configuration.

2. The console can only control whether the task itself is running, but cannot control the start and stop of the job process. Because the console and the job server are completely distributed, the console cannot control the job server.

The main function:

1. Check job and server status

2. Quickly modify and delete job configurations

3. Enable and disable jobs

4. View jobs across registration centers

5. View job running track and running status

Item not supported

1. Add a job. Because jobs are automatically added when running for the first time, it is not necessary to use the console to add jobs. Just start the job process containing Elasitc-Job directly on the job server.

8.2.1 Building steps

• Unzip elastic-job-lite-console-2.1.5.tar

• Enter the bin directory and execute:

  bin\start.bat
  

• Open the browser and visit http://localhost:8899 Username: root Password: root. After entering, the interface is as follows:

  

  Two types of users are provided: administrators and guests. Administrators have all operating rights, and guests only have viewing rights. The default administrator account and password are root/root, and the guest username and password are guest/guest. The administrator and guest usernames and passwords can be modified through conf\auth.properties.

8.2.2 Configuration and use

• To configure the registration center address, first start zookeeper and then go to the registration center configuration interface, click Add

  

• After clicking Submit, then click Connect (zookeeper must be started)

  

• After the connection is successful, information such as the namespace job name, number of shards, and the cron expression of the job can be displayed under the job latitude. Information such as the server IP, the number of currently running instances, and the total number of jobs can be viewed at the server latitude.

  

• After adding a database connection, you can view the execution results of the task

  

• Then you can see the task execution history in the job history.

  
  demo download address: https://download.csdn.net/download/zsx1314lovezyf/88282573

​

​