Integrated elastic-job distributed scheduling scheduled tasks

foreword

The timing task component is often used in the work process. On a stand-alone node, you can directly choose to use Spring ’s own timing task component hubble-task . Once this timing task is confirmed and solidified with a timing trigger strategy, it cannot be turned on and off dynamically. , so there was Quartz later .

Quartz is an open source project in the field of timed tasks. It is developed by JAVA . It can schedule the start, stop and strategy of timed tasks through API , and has powerful functions such as support for JTA transactions and clusters.

But Quartz has some of its disadvantages:

• Quartz adjusts timing tasks and needs to be scheduled through the API , which is not separated from the business system in essence.
• Quartz needs to persist data to the underlying data table, and the data intrusion into the business system is relatively high.
• Quartz also does not support distributed scheduling execution, only a single task can be executed individually

The elastic -job is Dangdang's second package on the basis of Quartz. There are two versions of elastic-job :

• Elastic-Job-Cloud : Deployment method for microservices
• Elastic-Job-Lite : Deployment based on zookeeper as a registration center

These two versions are the same in api except for the different deployment methods . Elastic-job has added many new features compared to Quartz :

1. Support UI page, you can dynamically adjust the timing strategy and start and stop on the web page
2. Based on Zookeeper as distributed scheduling, scheduling is decoupled from tasks
3. Supports distributed scheduling sharding, the same task can be divided into multiple pieces for execution
4. Various job types, supporting Simple, DataFLow data flow, and Script scripts
5. Failover, the tasks of offline machines will be re-sharded for execution
6. The consistency of job sharding, tasks will not be repeated after sharding
7. Job Compensation for Missed Executions

Install

To install elastic-job-lite , you need to install zookeeper in advance . If you need an installation tutorial, you can read this article: Install Zookeeper Online on Linux

The address of elastic-job in apache: elasticjob

Then you need to run the jar file containing Elastic-Job-Lite and business code. Not limited to jar or war startup methods.

Source address: elastic-job-lite

start elastic-job-lite-console

Download the source code of version 2.1.4:

https://codeload.github.com/apache/shardingsphere-elasticjob/zip/refs/tags/2.1.4

After the download is complete and decompressed, there are the following directories:

Enter the elastic-job-lite -console under the elastic-job-lite file . Package in this directory, package command

mvn clean install -Dmaven.test.skip=true  

After packaging, start the jar package, or start the source code directly, find the ConsoleBootstrap class under the console module to start

After the startup is complete, visit ip:8899 , and the account password is: root/root

After entering the system, enter the registration center configuration, fill in the zookeeper address that needs to be registered to connect.

The following is the installation configuration in linux . You can also directly put the packaged package into linux for execution.

install console

After elastic-job 3.0, there is no console module, and there is a more beautiful ui console

Download the console package directly here:

https://archive.apache.org/dist/shardingsphere/elasticjob-ui-3.0.0-RC1/apache-shardingsphere-elasticjob-3.0.0-RC1-lite-ui-bin.tar.gz

After downloading, upload to the server for decompression

 tar -zxvf apache-shardingsphere-elasticjob-3.0.0-RC1-lite-ui-bin.tar.gz

Go to the bin file to start

./start.sh

After successful startup, access the ip address: 8088 , and the default account password is root/root

After entering, you need to add the registration center in the global configuration - registration center configuration

After the addition is completed, the connection is established, and the elastic-job is initially set up. If you want to import the data source, you need to modify the application.properties configuration file under the conf file

I want to change it to mysql as the database, and I need to add the connection package to the lib file, which can be uploaded manually.

vim application.properties

Change to mysql driver and connection method

Save the file and restart elastic-job , add a data source in the event tracking data source configuration, as shown below:

Click to establish a connection, and the configuration and logs of scheduled tasks will be recorded in the table

integrated

easy integration

Introduce pom dependencies

<dependency>
	<groupId>com.cxytiandi</groupId>
	<artifactId>elastic-job-spring-boot-starter</artifactId>
	<version>1.0.0</version>
</dependency>
<dependency>
	<groupId>org.apache.curator</groupId>
	<artifactId>curator-framework</artifactId>
	<version>2.10.0</version>
</dependency>
<dependency>
	<groupId>org.apache.curator</groupId>
	<artifactId>curator-recipes</artifactId>
	<version>2.10.0</version>
</dependency>

Add configuration in the configuration file application.properties:

elasticJob.zk.serverLists=localhost:2181
elasticJob.zk.namespace=user-sync

Then you can directly define a job class to verify, the code is as follows:

@Component
@ElasticJobConf(name = "TestJob",cron="0 0 0 * * ?",shardingTotalCount=5) // 每天零点执行
public class TestJob extends SimpleJob{
    
    
  @Override
  public void execute(ShardingContext shardingContext) {
    
    
  	// 要执行的逻辑
  }
}

The timing strategy of this method depends on the ElasticJobConf annotation, just adjust the configuration of the annotation.

General integration

The general integration has three classes

• ElasticJobConfig : configuration of elastic-job components, for example zookeeper configuration center
• ElasticJobHandler : The specific execution class of the job task can be configured here
• ElasticJobListener : job task monitoring, start and end
• ElasticJobProperties : read zookeeper configuration from configuration file

Introduce dependencies

<dependency>
    <groupId>com.dangdang</groupId>
    <artifactId>elastic-job-common-core</artifactId>
    <version>${elasticjob.version}</version>
    <exclusions>
        <exclusion>
            <artifactId>guava</artifactId>
            <groupId>com.google.guava</groupId>
        </exclusion>
        <exclusion>
            <artifactId>curator-framework</artifactId>
            <groupId>org.apache.curator</groupId>
        </exclusion>
    </exclusions>
</dependency>
<dependency>
    <groupId>com.dangdang</groupId>
    <artifactId>elastic-job-lite-core</artifactId>
    <version>${elasticjob.version}</version>
</dependency>

<dependency>
    <groupId>com.dangdang</groupId>
    <artifactId>elastic-job-lite-spring</artifactId>
    <version>${elasticjob.version}</version>
</dependency>
<!--解决冲突 elasticjob-->
<dependency>
    <groupId>org.apache.curator</groupId>
    <artifactId>curator-recipes</artifactId>
    <version>2.10.0</version>
</dependency>

ElasticJobConfig

import com.dangdang.ddframe.job.reg.zookeeper.ZookeeperConfiguration;
import com.dangdang.ddframe.job.reg.zookeeper.ZookeeperRegistryCenter;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.context.annotation.Bean;
import org.springframework.stereotype.Component;

@Component
@ConditionalOnProperty(name = "elasticjob.enabled", havingValue = "true")
public class ElasticJobConfig {
    
    
    private final ElasticJobProperties jobProperties;

    public ElasticJobConfig(ElasticJobProperties jobProperties) {
    
    
        this.jobProperties = jobProperties;
    }

    @Bean(initMethod = "init")
    public ZookeeperRegistryCenter regCenter() {
    
    
        return new ZookeeperRegistryCenter(new ZookeeperConfiguration(jobProperties.getServerLists(),
                jobProperties.getNamespace()));
    }

    @Bean
    public ElasticJobListener elasticJobListener() {
    
    
        return new ElasticJobListener(100, 100);
    }
}

ElasticJobHandler

import com.dangdang.ddframe.job.api.simple.SimpleJob;
import com.dangdang.ddframe.job.config.JobCoreConfiguration;
import com.dangdang.ddframe.job.config.simple.SimpleJobConfiguration;
import com.dangdang.ddframe.job.lite.api.JobScheduler;
import com.dangdang.ddframe.job.lite.config.LiteJobConfiguration;
import com.dangdang.ddframe.job.lite.spring.api.SpringJobScheduler;
import com.dangdang.ddframe.job.reg.zookeeper.ZookeeperRegistryCenter;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.autoconfigure.AutoConfigureAfter;
import org.springframework.boot.autoconfigure.condition.ConditionalOnBean;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Slf4j
@Configuration
@AutoConfigureAfter(com.kyf.pe.trade.dataprocess.assist.config.ElasticJobConfig.class)
@ConditionalOnBean(com.kyf.pe.trade.dataprocess.assist.config.ElasticJobConfig.class)
public class ElasticJobHandlerConfig {
    
    
    private final ZookeeperRegistryCenter zookeeperRegistryCenter;

    public ElasticJobHandlerConfig(ZookeeperRegistryCenter zookeeperRegistryCenter) {
    
    
        this.zookeeperRegistryCenter = zookeeperRegistryCenter;
    }

    /**
     * 配置任务详细信息
     *
     * @param jobClass               定时任务实现类
     * @param cron                   表达式
     * @param shardingTotalCount     分片数
     * @param shardingItemParameters 分片参数
     * @return
     */
    private LiteJobConfiguration getLiteJobConfiguration(final Class<? extends SimpleJob> jobClass,
                                                         final String cron,
                                                         final int shardingTotalCount,
                                                         final String shardingItemParameters,
                                                         final String jobParameters,
                                                         final String description) {
    
    
        // 定义作业核心配置
        JobCoreConfiguration simpleCoreConfig = JobCoreConfiguration.newBuilder(jobClass.getSimpleName(), cron, shardingTotalCount).
                shardingItemParameters(shardingItemParameters).jobParameter(jobParameters).description(description).build();
        // 定义SIMPLE类型配置
        SimpleJobConfiguration simpleJobConfig = new SimpleJobConfiguration(simpleCoreConfig, jobClass.getCanonicalName());
        // 定义Lite作业根配置
        return LiteJobConfiguration.newBuilder(simpleJobConfig).build();
    }


    /**
     * 具体任务
     */
    @Bean(initMethod = "init")
    public JobScheduler pushHrhbJobScheduler(final TestJob testjob,
                                             @Value("${job.test.cron}") final String cron,
                                             @Value("${job.test.shardingTotalCount}") final int shardingTotalCount,
                                             @Value("${job.test.description}") final String description) {
    
    

        return new SpringJobScheduler(testjob, zookeeperRegistryCenter, getLiteJobConfiguration(testjob.getClass(),
                cron, shardingTotalCount, "", "", description));
    }


}

ElasticJobListener

import com.dangdang.ddframe.job.executor.ShardingContexts;
import com.dangdang.ddframe.job.lite.api.listener.AbstractDistributeOnceElasticJobListener;
import lombok.extern.slf4j.Slf4j;

@Slf4j
public class ElasticJobListener extends AbstractDistributeOnceElasticJobListener {
    
    

    /**
     * 设置间隔时间
     *
     * @param startedTimeoutMilliseconds
     * @param completedTimeoutMilliseconds
     */
    public ElasticJobListener(long startedTimeoutMilliseconds, long completedTimeoutMilliseconds) {
    
    
        super(startedTimeoutMilliseconds, completedTimeoutMilliseconds);
    }

    @Override
    public void doBeforeJobExecutedAtLastStarted(ShardingContexts shardingContexts) {
    
    
        log.info("任务名：{}开始", shardingContexts.getJobParameter());
    }

    @Override
    public void doAfterJobExecutedAtLastCompleted(ShardingContexts shardingContexts) {
    
    
        log.info("任务名：{}结束", shardingContexts.getJobParameter());
    }
}

ElasticJobProperties

import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.context.annotation.Configuration;

@Configuration
@ConfigurationProperties(prefix = "elasticjob")
public class ElasticJobProperties {
    
    
    private boolean enabled = true;

    private String serverLists;

    private String namespace;

    public boolean isEnabled() {
    
    
        return enabled;
    }

    public void setEnabled(boolean enabled) {
    
    
        this.enabled = enabled;
    }

    public String getServerLists() {
    
    
        return serverLists;
    }

    public void setServerLists(String serverLists) {
    
    
        this.serverLists = serverLists;
    }

    public String getNamespace() {
    
    
        return namespace;
    }

    public void setNamespace(String namespace) {
    
    
        this.namespace = namespace;
    }
}