3,000 words take you to understand the XXL-JOB task scheduling platform

mind Mapping

The article has been included in Github selection, welcome to Star : https://github.com/yehongzhi/learningSummary

I. Overview

In normal business scenarios, there are often some scenarios that need to use timed tasks, such as:

• Time-driven scenario: send coupons at a certain point in time, send SMS, etc.
• Batch processing data: batch statistics of last month's bills, statistics of last month's sales data, etc.
• Fixed frequency scenario: It needs to be executed every 5 minutes.

Therefore, timed tasks are not uncommon in normal development, and for the current era of rapid consumption, various pushes need to be sent every day, and messages need to be completed by timed tasks, which is very widely used.

2. Why do you need a task scheduling platform

In Java, traditional timing task implementation solutions, such as Timer, Quartz, etc., have more or less problems:

• No cluster support, no statistics, no management platform, no failure alarm, no monitoring, etc.

And in the current distributed architecture, there are some scenarios that require distributed task scheduling:

• When tasks of multiple instances of the same service are mutually exclusive, unified scheduling is required.
• Task scheduling needs to support high availability, monitoring, and fault alarms.
• It is necessary to uniformly manage and track the results of task scheduling of each service node, and it is necessary to record and save task attribute information.

Obviously, traditional timing tasks are no longer sufficient for the current distributed architecture, so a distributed task scheduling platform is needed. At present, the more mainstream ones are elasticjob and xxl-job.

Elasticjob is open sourced by Dangdang. Currently, github has 6.5k Star, and 76 companies are registered on the official website.

Different from xxl-job, elasticjob uses zookeeper to implement distributed coordination and achieve high task availability and fragmentation.

Three, why choose XXL-JOB

In fact, more companies choose xxl-job. Currently , there are 15.7k stars on xxl-job's github, and there are 348 registered companies . There is no doubt that both elasticjob and xxl-job are very excellent technical frameworks. Next, we will further compare and discuss and explore why more companies choose xxl-job.

First introduce xxl-job. This is an open source project from Dianping Xu Xueli (xxl is the first letter of the author’s name). The official website introduces this as a lightweight distributed task scheduling framework. Its core design goal is rapid development. , Easy to learn, lightweight, and easy to expand. Unlike elasticjob, the xxl-job environment relies on mysql instead of ZooKeeper, which is also the biggest difference.

The original intention of elasticjob is to face complex services with high concurrency, even when the business volume is large and there are many servers, it can do a good job scheduling, and use server resources as much as possible. Use ZooKeeper to make it highly available, consistent, and scalable. The elasticjob written on the official website is decentralized. The main server is elected through ZooKeeper's election mechanism. If the main server fails, a new main server will be re-elected. Therefore, elasticjob has good scalability and availability, but its use and operation and maintenance are somewhat complicated .

xxl-job is the opposite. It uses a centralized scheduling platform to schedule multiple executors to perform tasks. The scheduling center uses DB locks to ensure the consistency of cluster distributed scheduling, so that expanding executors will increase the pressure on DB, but if In fact, the database here is only responsible for the scheduling and execution of tasks. But without a large number of executors and tasks, it will not cause database pressure. In fact, most companies do not have many executors (although interviews often ask some high-concurrency questions).

Relatively speaking, the xxl-job centralized scheduling platform is lightweight, out -of-the- box, easy to operate, quick to get started, and has a very good integration with SpringBoot , and the monitoring interface is integrated in the scheduling center, and the interface is simple and convenient for enterprises The maintenance cost is not high, and there are failed email alerts and so on. This makes many companies choose xxl-job as a scheduling platform.

Four, installation

4.1 Pull the source code

Setting up xxl-job is very simple. There are two ways of docker pull image deployment and source code compilation. The way of docker deployment is relatively simple. I will talk about source code compilation. First go to github to pull the source code of xxl-job to the local.

4.2 Import IDEA

After pulling down the source code, you can see the project structure as follows:

Import it into IDEA, configure Maven, download the relevant jar package, after a while, you can see such a project:

4.3 Initialize the database

As mentioned earlier, xxl-job needs to rely on mysql, so the database needs to be initialized. Find the tables_xxl_job.sql file under the xxl-jobdocdb path. Run the sql file on mysql.

4.4 Configuration file

Then change the configuration file and find the application.properties file under the admin project.

### 调度中心JDBC链接
spring.datasource.url=jdbc:mysql://127.0.0.1:3306/xxl_job?useUnicode=true&characterEncoding=UTF-8&autoReconnect=true&serverTimezone=Asia/Shanghai
spring.datasource.username=root
spring.datasource.password=
spring.datasource.driver-class-name=com.mysql.jdbc.Driver
### 报警邮箱
spring.mail.host=smtp.qq.com
spring.mail.port=25
[email protected]
spring.mail.password=xxx
spring.mail.properties.mail.smtp.auth=true
spring.mail.properties.mail.smtp.starttls.enable=true
spring.mail.properties.mail.smtp.starttls.required=true
spring.mail.properties.mail.smtp.socketFactory.class=javax.net.ssl.SSLSocketFactory
### 调度中心通讯TOKEN [选填]：非空时启用；
xxl.job.accessToken=
### 调度中心国际化配置 [必填]： 默认为 "zh_CN"/中文简体, 可选范围为 "zh_CN"/中文简体, "zh_TC"/中文繁体 and "en"/英文；
xxl.job.i18n=zh_CN
## 调度线程池最大线程配置【必填】
xxl.job.triggerpool.fast.max=200
xxl.job.triggerpool.slow.max=100
### 调度中心日志表数据保存天数 [必填]：过期日志自动清理；限制大于等于7时生效，否则, 如-1，关闭自动清理功能；
xxl.job.logretentiondays=10

4.5 Compile and run

Simply run the main method of the admin project to start it.

If it is deployed on the server, then we need to package it into a jar package and use the Maven plugin to package it in IDEA.

Then find the jar package in the xxl-jobxxl-job-admintarget path.

Then you get the jar package, and you can start it with the java -jar command.

It's done here! Open the browser and enter http://localhost:8080/xxl-job-admin to enter the management page. Default account/password: admin/123456.

Five, forever HelloWord

After deploying the dispatch center, you need to register the executor with the dispatch center and add scheduling tasks. Next, write a simple example with reference to xxl-job.

First create a SpringBoot project called "xxljob-demo" and add dependencies.

<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter</artifactId>
    </dependency>
    <dependency><!-- 官网的demo是2.2.1，中央maven仓库还没有，所以就用2.2.0 -->
        <groupId>com.xuxueli</groupId>
        <artifactId>xxl-job-core</artifactId>
        <version>2.2.0</version>
    </dependency>
</dependencies>

Then modify application.properties.

# web port
server.port=8081
# log config
logging.config=classpath:logback.xml
spring.application.name=xxljob-demo
### 调度中心部署跟地址 [选填]：如调度中心集群部署存在多个地址则用逗号分隔。执行器将会使用该地址进行"执行器心跳注册"和"任务结果回调"；为空则关闭自动注册；
xxl.job.admin.addresses=http://127.0.0.1:8080/xxl-job-admin
### 执行器通讯TOKEN [选填]：非空时启用；
xxl.job.accessToken=
### 执行器AppName [选填]：执行器心跳注册分组依据；为空则关闭自动注册
xxl.job.executor.appname=xxl-job-demo
### 执行器注册 [选填]：优先使用该配置作为注册地址，为空时使用内嵌服务 ”IP:PORT“ 作为注册地址。从而更灵活的支持容器类型执行器动态IP和动态映射端口问题。
xxl.job.executor.address=
### 执行器IP [选填]：默认为空表示自动获取IP，多网卡时可手动设置指定IP，该IP不会绑定Host仅作为通讯实用；地址信息用于 "执行器注册" 和 "调度中心请求并触发任务"；
xxl.job.executor.ip=
### 执行器端口号 [选填]：小于等于0则自动获取；默认端口为9999，单机部署多个执行器时，注意要配置不同执行器端口；
xxl.job.executor.port=9999
### 执行器运行日志文件存储磁盘路径 [选填] ：需要对该路径拥有读写权限；为空则使用默认路径；
xxl.job.executor.logpath=/data/applogs/xxl-job/jobhandler
### 执行器日志文件保存天数 [选填] ： 过期日志自动清理, 限制值大于等于3时生效; 否则, 如-1, 关闭自动清理功能；
xxl.job.executor.logretentiondays=10

Then write a configuration class XxlJobConfig.

@Configuration
public class XxlJobConfig {
    private Logger logger = LoggerFactory.getLogger(XxlJobConfig.class);
    @Value("${xxl.job.admin.addresses}")
    private String adminAddresses;
    @Value("${xxl.job.accessToken}")
    private String accessToken;
    @Value("${xxl.job.executor.appname}")
    private String appname;
    @Value("${xxl.job.executor.address}")
    private String address;
    @Value("${xxl.job.executor.ip}")
    private String ip;
    @Value("${xxl.job.executor.port}")
    private int port;
    @Value("${xxl.job.executor.logpath}")
    private String logPath;
    @Value("${xxl.job.executor.logretentiondays}")
    private int logRetentionDays;

    @Bean
    public XxlJobSpringExecutor xxlJobExecutor() {
        logger.info(">>>>>>>>>>> xxl-job config init.");
        XxlJobSpringExecutor xxlJobSpringExecutor = new XxlJobSpringExecutor();
        xxlJobSpringExecutor.setAdminAddresses(adminAddresses);
        xxlJobSpringExecutor.setAppname(appname);
        xxlJobSpringExecutor.setAddress(address);
        xxlJobSpringExecutor.setIp(ip);
        xxlJobSpringExecutor.setPort(port);
        xxlJobSpringExecutor.setAccessToken(accessToken);
        xxlJobSpringExecutor.setLogPath(logPath);
        xxlJobSpringExecutor.setLogRetentionDays(logRetentionDays);
        return xxlJobSpringExecutor;
    }
}

Then write a task class XxlJobDemoHandler, using Bean mode.

@Component
public class XxlJobDemoHandler {
    /**
     * Bean模式，一个方法为一个任务
     * 1、在Spring Bean实例中，开发Job方法，方式格式要求为 "public ReturnT<String> execute(String param)"
     * 2、为Job方法添加注解 "@XxlJob(value="自定义jobhandler名称", init = "JobHandler初始化方法", destroy = "JobHandler销毁方法")"，注解value值对应的是调度中心新建任务的JobHandler属性的值。
     * 3、执行日志：需要通过 "XxlJobLogger.log" 打印执行日志；
     */
    @XxlJob("demoJobHandler")
    public ReturnT<String> demoJobHandler(String param) throws Exception {
        XxlJobLogger.log("java, Hello World~~~");
        XxlJobLogger.log("param:" + param);
        return ReturnT.SUCCESS;
    }
}

In the resources directory, add the logback.xml file.

<?xml version="1.0" encoding="UTF-8"?>
<configuration debug="false" scan="true" scanPeriod="1 seconds">
    <contextName>logback</contextName>
    <property name="log.path" value="/data/applogs/xxl-job/xxl-job-executor-sample-springboot.log"/>
    <appender name="console" class="ch.qos.logback.core.ConsoleAppender">
        <encoder>
            <pattern>%d{HH:mm:ss.SSS} %contextName [%thread] %-5level %logger{36} - %msg%n</pattern>
        </encoder>
    </appender>
    <appender name="file" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>${log.path}</file>
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>${log.path}.%d{yyyy-MM-dd}.zip</fileNamePattern>
        </rollingPolicy>
        <encoder>
            <pattern>%date %level [%thread] %logger{36} [%file : %line] %msg%n
            </pattern>
        </encoder>
    </appender>
    <root level="info">
        <appender-ref ref="console"/>
        <appender-ref ref="file"/>
    </root>
</configuration>

After writing, start the service, then you can open the management interface, find the actuator management, and add the actuator.

Then go to task management and add tasks.


Finally, we can go to task management to test and run demoJobHandler.


After clicking save, it will be executed immediately. Click View log, you can see the historical log record of task execution.

Open the execution log just executed, we can see that the operation was successful.

This is a simple Demo demonstration, very simple and quick to get started.

Six, talk about architecture design

Let's briefly talk about the architecture of xxl-job. Let's first look at an architecture diagram provided by the official website for analysis.

As can be seen from the architecture diagram, there are two major components of the dispatch center and the actuator.

• Dispatch center. Responsible for managing scheduling information , sending scheduling requests in accordance with the scheduling configuration, and not responsible for business codes. Support visual interface, you can add, update, and delete tasks in the dispatch center, which will take effect in real time. Support monitoring scheduling results, viewing execution logs, viewing scheduling task statistical reports, task failure alarms, etc.
• Actuator. Responsible for receiving scheduling requests and executing the business logic of scheduling tasks. After the actuator is started, it needs to be registered to the dispatch center. Receive execution requests, termination requests, log requests and so on from the dispatch center.

Next we look at the working principle of xxl-job.

• The task executor is automatically registered to the dispatch center according to the address of the dispatch center configured.
• When the task trigger condition is reached, the dispatch center issues the task.
• The executor executes tasks based on the thread pool, puts the execution results in the memory queue, and writes the execution log into the log file.
• The executor's callback thread consumes the execution result in the memory queue and actively reports it to the dispatch center.
• When the user views the task log in the dispatch center, the dispatch center requests the task executor, and the task executor reads the task log file and returns the log details.

Talk

After reading the above content, it is basically an introduction. In fact, xxl-job still has many functions. To learn in depth, you also need to go to the official website to study and explore. The best way is to build an xxl-job locally to play and play, hands-on practice is the fastest way to learn.

 

Original link
This article is the original content of Alibaba Cloud and may not be reproduced without permission.