[] Distributed distributed task scheduling platform

I. Overview
- What is the timing task
Two, Java way to achieve timing tasks
Third, what issues will be regular tasks distributed situation?
Fourth, regular tasks distributed solutions
Five, XXLJOB introduction

I. Overview

What is the timing task

Timing task scheduling (development) corresponding to a certain time period, some task operation. A timing (a time period for operation at 2 points is assumed) Task (demand) schedule (trigger)
Case: Timing task scheduling cases, every morning I need to know the amount of users active day yesterday, writes a regular Job, 9 o'clock every morning, day inquiry yesterday UI users live monthly amount, sent to my mailbox by mail.
Timing task scenarios: data synchronization, transaction information, clear user information, regularly sends report data, events push
Leads to a problem: the distributed task scheduling platform in the field of distributed server cluster, then, how to ensure that the timing of Job idempotency

Two, Java way to achieve timing tasks

2.1 Thread

public class Demo01 {
    static long count = 0;
    public static void main(String[] args) {
        Runnable runnable = new Runnable() {
            @Override
            public void run() {
                while (true) {
                    try {
                        Thread.sleep(1000);
                        count++;
                        System.out.println(count);
                    } catch (Exception e) {
                        // TODO: handle exception
                    }
                }
            }
        };
        Thread thread = new Thread(runnable);
        thread.start();
    }
}

2.2 TimerTask

/**
 * 使用TimerTask类实现定时任务
*/
public class Demo02 {
    static long count = 0;
    public static void main(String[] args) {
        TimerTask timerTask = new TimerTask() {
            @Override
            public void run() {
            count++;
            System.out.println(count);
        }
    };
    Timer timer = new Timer();
    // 天数
    long delay = 0;
    // 秒数
    long period = 1000;
    timer.scheduleAtFixedRate(timerTask, delay, period);
}

}

2.3 ScheduledExecutorService

ScheduledExecutorService from using the Java
JavaSE5 of java.util.concurrent, the tools to be as complicated by the introduction, this is the ideal way to achieve the scheduled task.

public class Demo003 {
    public static void main(String[] args) {
        Runnable runnable = new Runnable() {
            public void run() {
                // task to run goes here
                System.out.println("Hello !!");
            }
        };
        ScheduledExecutorService service = Executors.newSingleThreadScheduledExecutor();
        // 第二个参数为首次执行的延时时间，第三个参数为定时执行的间隔时间
        service.scheduleAtFixedRate(runnable, 1, 1, TimeUnit.SECONDS);
    }
}

2.4 Quartz

Introduced maven dependence

<dependencies>
    <!-- quartz -->
    <dependency>
        <groupId>org.quartz-scheduler</groupId>
        <artifactId>quartz</artifactId>
        <version>2.2.1</version>
    </dependency>
    <dependency>
        <groupId>org.quartz-scheduler</groupId>
        <artifactId>quartz-jobs</artifactId>
        <version>2.2.1</version>
    </dependency>
</dependencies>

Task scheduling classes

public class MyJob implements Job {
    public void execute(JobExecutionContext context) throws JobExecutionException {
        System.out.println("quartz MyJob date:" + new Date().getTime());
    }
}

Startup class

 //1.创建Scheduler的工厂
  SchedulerFactory sf = new StdSchedulerFactory();
  //2.从工厂中获取调度器实例
  Scheduler scheduler = sf.getScheduler();

  //3.创建JobDetail
  JobDetail jb = JobBuilder.newJob(MyJob.class)
          .withDescription("this is a ram job") //job的描述
          .withIdentity("ramJob", "ramGroup") //job 的name和group
          .build();

  //任务运行的时间，SimpleSchedle类型触发器有效
  long time=  System.currentTimeMillis() + 3*1000L; //3秒后启动任务
  Date statTime = new Date(time);

  //4.创建Trigger
      //使用SimpleScheduleBuilder或者CronScheduleBuilder
  Trigger t = TriggerBuilder.newTrigger()
              .withDescription("")
              .withIdentity("ramTrigger", "ramTriggerGroup")
              //.withSchedule(SimpleScheduleBuilder.simpleSchedule())
              .startAt(statTime)  //默认当前时间启动
              .withSchedule(CronScheduleBuilder.cronSchedule("0/2 * * * * ?")) //两秒执行一次
              .build();

  //5.注册任务和定时器
  scheduler.scheduleJob(jb, t);

  //6.启动 调度器
  scheduler.start();

Quartz expression

http://cron.qqe2.com/

Third, what issues will be regular tasks distributed situation?

In the case of a distributed cluster, how to ensure the implementation of scheduled tasks will not be repeated

Regular tasks and business code stored in the same jvm (small project)
Large Internet companies scheduled task code execution and execute code server operations are separate, are independent jvm.
Whether regular tasks need to be considered highly concurrent server? Points to perform multiple tasks at the same time interval scene is not required, and high concurrency scenarios may occur
How then if in the case of high concurrency, regular downtime should deal with Job? (Single node) using a heartbeat monitor automatically restart, compensation mechanisms (each task to make a small mark) regular tasks suddenly in the middle of error when executing code the use of logging the error, skipped to continue, using the timing error JOb scan log record information to compensate. Timing] ob at the time of execution, resulting in the Job aborts the send e-mail notification to the operation and maintenance personnel

Traditional distributed tasks and the timing difference between the timing task
conventional scheduled task characteristics: single (no clusters)

Fourth, regular tasks distributed solutions

Use zookeeper implement a distributed lock disadvantages (need to create a temporary node, and event notification is not easy to expand)
Use the configuration file to make a switch after the release of shortcomings need to reboot, set up a task switch is turned on, will be one of the servers is set to true, on behalf of open tasks, other servers set to false, this way if the server is set to true down, the task becomes ineffective. (Not recommended)
The only constraint database, the disadvantage of low efficiency (not recommended)
Using a distributed task scheduling platform XXLJOB, Elastric-Job, TBSchedule

Five, XXLJOB introduction

5.1 Distributed scheduling platform can help us achieve those things

Support Job cluster (the premise of guaranteeing idempotency issues) Job load balancing mechanism in rotation
Support Job Job execution compensation if it fails, it will automatically retry mechanism if it fails to restart repeatedly or send e-mail notification to the operation and maintenance personnel.
Support Job Logging
Dynamic configuration timing rules of traditional timed JOb trigger rules are written dead, dynamic configuration Job Rules

5.2 XXLJOB GitHub

XXLJOB project and documentation

5.3 Principles

XXL-Job Principle: execution, Task Manager
- Actuator expression meaning: regular services address the practical implementation of the Job
- Task tubules meaning: a timing task configuration rules, routing policy, permission mode, and the like.

step

① deployment: xxl-job-admin as a registration center
② creation actuator (scheduling specific address) can support a cluster
③ profiles need to fill xxl-job registry address
④ each specific job execution server to establish the connection port number a netty
⑤ need to perform job tasks class, integrated IJobHandler abstract class registration to the job container
⑥ Execute method to prepare specific job tasks

5.4 SpringBoot integration XXLJob

Profile Information

application.properties

# web port
server.port=8081
    
# log config
logging.config=classpath:logback.xml
    
### xxl-job admin address list, such as "http://address" or "http://address01,http://address02"
xxl.job.admin.addresses=http://127.0.0.1:8080/xxl-job-admin
    
### xxl-job executor address
xxl.job.executor.appname=text-job
xxl.job.executor.ip=
xxl.job.executor.port=9999
    
### xxl-job, access token
xxl.job.accessToken=
    
### xxl-job log path
xxl.job.executor.logpath=/data/applogs/xxl-job/jobhandler
### xxl-job log retention days
xxl.job.executor.logretentiondays=-1

Configuration XxlJobConfig

@Configuration
@ComponentScan(basePackages = "com.xxl.job.executor.service.jobhandler")
public class XxlJobConfig {
    private Logger logger = LoggerFactory.getLogger(XxlJobConfig.class);

@Value("${xxl.job.admin.addresses}")
private String adminAddresses;

@Value("${xxl.job.executor.appname}")
private String appName;

@Value("${xxl.job.executor.ip}")
private String ip;

@Value("${xxl.job.executor.port}")
private int port;

@Value("${xxl.job.accessToken}")
private String accessToken;

@Value("${xxl.job.executor.logpath}")
private String logPath;

@Value("${xxl.job.executor.logretentiondays}")
private int logRetentionDays;

@Bean(initMethod = "start", destroyMethod = "destroy")
public XxlJobExecutor xxlJobExecutor() {
    logger.info(">>>>>>>>>>> xxl-job config init.");
    XxlJobExecutor xxlJobExecutor = new XxlJobExecutor();
    xxlJobExecutor.setAdminAddresses(adminAddresses);
    xxlJobExecutor.setAppName(appName);
    xxlJobExecutor.setIp(ip);
    xxlJobExecutor.setPort(port);
    xxlJobExecutor.setAccessToken(accessToken);
    xxlJobExecutor.setLogPath(logPath);
    xxlJobExecutor.setLogRetentionDays(logRetentionDays);
    return xxlJobExecutor;
}
}

Create a handler interfaces

@JobHandler("demoJobHandler")
@Component
public class DemoHandler extends IJobHandler {
    @Value("${server.port}")
    private String serverPort;
    
    @Override
    public ReturnT<String> execute(String param) throws Exception {
        System.out.println("######端口号：serverPort" + serverPort + "###定时Job开始执行啦!!!!######");
        return SUCCESS;
    }
}

5.5 dispatch center cluster

　　If xx-job admin platform hang up, then, will lead the task can not be performed, so the dispatch center also need to deploy cluster dispatch center supports cluster deployment, improve scheduling system disaster recovery and availability. When the dispatch center cluster deployment, several requirements and recommendations:

DB configuration consistent;
Login account configuration consistency;
Group machine clock consistent (single cluster neglect);
Recommendation: recommended to do load balancing cluster dispatch center by nginx, assignment of domain names. Dispatch center access, actuators configured callback, call the API service and other operations are carried out by the domain name.

Nginx configuration information

upstream  backServer{
server 127.0.0.1:8080 weight=1;
server 127.0.0.1:8081 weight=1;
}
server {
    listen       80;
    server_name  127.0.0.1 ;
    #charset koi8-r;
    #access_log  logs/host.access.log  main;
    location / {
        proxy_pass   http://backServer;
        index  index.html index.htm;
    }
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   html;
    }
}