Table of contents
1: Introduction to XXL-JOB
XXL-JOB is a lightweight distributed task scheduling platform. Its core design goals are rapid development, easy learning, lightweight, and easy expansion. The source code is now open and connected to the online product lines of many companies, out of the box.
Official website: https://www.xuxueli.com/xxl-job/
Documentation: https://www.xuxueli.com/xxl-job
XXL-JOB mainly includes scheduling center, executor, and tasks:
Dispatch Center:
Responsible for managing scheduling information, sending scheduling requests according to scheduling configuration, and not responsible for business codes;
The main responsibilities are executor management, task management, monitoring operation and maintenance, log management, etc.
Task Executor:
Responsible for receiving scheduling requests and executing task logic;
As long as the responsibilities are registration services, task execution services (task queues that will be placed in the thread pool after receiving tasks), execution result reporting, log services, etc.
Task: responsible for executing specific business processing.
The workflow between the dispatch center and the executor is as follows:
Implementation process:
1. The task executor automatically registers with the dispatch center according to the configured dispatch center address
2. When the task trigger condition is met, the dispatch center will issue the task
3. The executor executes the task based on the thread pool, puts the execution result into the memory queue, and writes the execution log into the log file
4. The executor consumes the execution results in the memory queue and actively reports to the scheduling center
5. When the user checks the task log in the dispatch center, the dispatch center requests the task executor, and the task executor reads the task log file and returns the log details
2: Build XXL-JOB
2.1: dispatch center
First download XXL-JOB
The project uses version 2.3.1: https://github.com/xuxueli/xxl-job/releases/tag/2.3.1
It can also be obtained from the course material directory, unzip xxl-job-2.3.1.zip
Use IDEA to open the decompressed directory
xxl-job-admin: scheduling center
xxl-job-core: public dependencies
xxl-job-executor-samples: Executor Sample (select the appropriate version of the executor and use it directly)
:xxl-job-executor-sample-springboot: Springboot version, manage the executor through Springboot, this method is recommended;
:xxl-job-executor-sample-frameless: frameless version;
doc : documentation, including database scripts
The xxl_job_2.3.1 database has been created in the MySQL of the distributed virtual machine
As shown below:
Account and password: admin/123456
If you cannot use the virtual machine to run xxl-job, you can run the xxl-job dispatch center on the local idea.
2.2: Actuator
The executor is configured below, and the executor is responsible for communicating with the dispatch center to receive the task dispatch request initiated by the dispatch center.
1. Enter the scheduling center below to add actuators
Click Add and fill in the executor information. appname is the application name of the executor specified when configuring the xxl information in nacos.
Added successfully: 
2: Add related dependencies to modules that need to use task scheduling in your project
<dependency>
<groupId>com.xuxueli</groupId>
<artifactId>xxl-job-core</artifactId>
</dependency>3: Add relevant configuration in the yaml configuration file
xxl:
job:
admin:
addresses: http://192.168.101.65:8088/xxl-job-admin
executor:
appname: media-process-service
address:
ip:
port: 9999
logpath: /data/applogs/xxl-job/jobhandler
logretentiondays: 30
accessToken: default_tokenNote that the appname in the configuration is the application name of the executor, and the port is the port on which the executor is started. If multiple executors are started locally, the ports cannot be repeated.
4. Configure the executor of xxl-job
Copy the configuration class under the xxl-job sample project to the service project of media asset management
Copy to: Modules that need to use task scheduling in their own projects
At this point, the service project configuration of the media asset management module is completed. The xxl-job executor is added, and the executor is added to the xxl-job dispatch center. Next, we are going to test whether the executor communicates with the dispatch center normally. Because the interface project depends on the service project, start the media asset The interface project of the management module.
Observe the log after startup, and the following log appears, indicating that the executor has successfully registered in the dispatch center
Simultaneously observe the executor interface in the dispatch center
1 actuator has been shown at the online machine address.
2.3: Executing tasks
To write the task below, refer to the writing method of the task class in the sample project, as shown in the figure below:
Add tasks in the dispatch center below and enter task management
Click Add and fill in the task information
Note the red mark:
Scheduling type:
Fixed speed refers to timing scheduling at fixed intervals.
Cron implements richer timing scheduling strategies through Cron expressions.
A cron expression is a string through which a scheduling policy can be defined, in the following format:
{seconds} {minutes} {hours} {date} {month} {week} {year (can be empty)}
xxl-job provides a graphical interface to configure:
Some examples are as follows:
30 10 1 * * ? Triggered at 1:10:30 every day
0/30 * * * * ? Fires every 30 seconds
* 0/10 * * * ? Fires every 10 minutes
There are BEAN and GLUE operating modes. The bean mode is more commonly used to write the task code of the executor in the project engineering. GLUE is to write the task code in the dispatch center.
JobHandler is the name of the task method, fill in the name in the @XxlJob annotation above the task method.
Routing strategy: When the executor cluster is deployed, which executor the dispatch center will deliver the task to. Selecting the first one here means only sending the task to the first executor. Other options of the routing strategy will be detailed in the fragment broadcast section later. explain.
Other configuration items of advanced configuration will be explained in detail later in the Fragment Broadcasting chapter.
Added successfully, start the task
View task execution status through scheduling logs
Next, start the service project of media asset management and start the executor.
Watch the execution of the executor method.
If you want to stop the task, you need to operate in the dispatch center
The task runs for a period of time and pays attention to clearing the log
3: Fragmented Broadcasting
After mastering the basic use of xxl-job, let's think about how to process distributed tasks? As shown in the figure below, we will start multiple executors to form a cluster to perform tasks.
What routing strategies does the scheduling center have when the executor is deployed in the cluster?
Check out the official xxl-job documentation to read about advanced configuration:
Advanced configuration:
- Routing strategy: When the executor cluster is deployed, a rich routing strategy is provided, including;
FIRST (first): fixedly select the first machine;
LAST (last): fixedly select the last machine;
ROUND ( Polling):
RANDOM (random): randomly select online machines;
CONSISTENT_HASH (consistency HASH): each task selects a certain machine according to the Hash algorithm, and all tasks are uniformly hashed on different machines.
LEAST_FREQUENTLY_USED (least frequently used): the machine with the lowest frequency of use is elected first;
LEAST_RECENTLY_USED (most recently unused): the machine that has not been used for the longest time is elected first;
FAILOVER (failover): heartbeat detection is performed in sequence, the first The machine with successful heartbeat detection is selected as the target executor and initiates scheduling;
BUSYOVER (busy transfer): idle detection is performed sequentially, and the first machine with successful idle detection is selected as the target executor and initiates scheduling;
SHARDING_BROADCAST (sharding Broadcast): Broadcast triggers all machines in the corresponding cluster to execute a task, and the system automatically transmits fragmentation parameters; fragmentation tasks can be developed according to fragmentation parameters; - Subtasks: each task has a unique task ID (task ID can
be Obtained from the task list), when the execution of this task is completed and the execution is successful, an active scheduling of the task corresponding to the subtask ID will be triggered. Through subtasks, one task can be executed to execute another task.
- Scheduling expiration policy:
- Ignore: After the schedule expires, ignore the expired task, and recalculate the next trigger time from the current time;
- Execute once immediately: After the schedule expires, execute it once immediately, and recalculate the next trigger time from the current time;
- Block Processing strategy: the processing strategy when the scheduling is too intensive and the executor is too late to process;
single-machine serial (default): after the scheduling request enters the single-machine executor, the scheduling request enters the FIFO queue and runs in serial;
discard subsequent scheduling: the scheduling request enters the single-machine After the executor, if it is found that there is a running scheduling task on the executor, this request will be discarded and marked as failed; scheduling
before coverage: After the scheduling request enters the stand-alone executor, it is found that the executor has a running scheduling task, and the operation will be terminated Scheduling tasks in and clear the queue, and then run local scheduling tasks;
- Task timeout: support custom task timeout, task running timeout will actively interrupt the task;
- Number of failed retries; support custom task failure retry times, When the task fails, it will actively retry according to the preset number of failed retries;
The next thing to focus on is the sharding broadcast strategy. Sharding means that the scheduling center slices the executors as the dimension, and marks the executors in the cluster with serial numbers: 0, 1, 2, 3..., broadcasting is It means that each scheduling will send task scheduling to all executors in the cluster, and the request will carry fragmentation parameters.
As shown below:
Each executor receives the scheduling request and receives the slice parameters at the same time.
xxl-job supports dynamic expansion of the executor cluster to dynamically increase the number of shards. When the amount of tasks increases, more executors can be deployed to the cluster, and the scheduling center will dynamically modify the number of shards.
What scenarios are job slices suitable for?
- Sharding task scenario: a cluster of 10 executors processes 100,000 pieces of data, each machine only needs to process 10,000 pieces of data, and the time-consuming is reduced by 10 times;
- Broadcast task scenario: The broadcast executor runs shell scripts at the same time, broadcasts cluster nodes for cache updates, etc.
Therefore, the broadcast sharding method can not only give full play to the capabilities of each executor, but also control whether the task is executed according to the sharding parameters, and finally flexibly control the distributed processing tasks of the executor cluster.
Instructions for use:
"Shard broadcast" is consistent with the normal task development process, the difference is that the slice parameters can be obtained for slice business processing.
Ways to obtain fragmentation parameters for Java language tasks:
BEAN, GLUE mode (Java), please refer to the sample task "ShardingJobHandler" in the Sample sample executor:
/**
* 2、分片广播任务
*/
@XxlJob("shardingJobHandler")
public void shardingJobHandler() throws Exception {
// 分片序号，从0开始
int shardIndex = XxlJobHelper.getShardIndex();
// 分片总数
int shardTotal = XxlJobHelper.getShardTotal();
....
The following test job fragmentation:
1. Define the task method of job sharding
/**
* 2、分片广播任务
*/
@XxlJob("shardingJobHandler")
public void shardingJobHandler() throws Exception {
// 分片参数
int shardIndex = XxlJobHelper.getShardIndex();
int shardTotal = XxlJobHelper.getShardTotal();
log.info("分片参数：当前分片序号 = {}, 总分片数 = {}", shardIndex, shardTotal);
log.info("开始执行第"+shardIndex+"批任务");
}2. Add tasks in the dispatch center
Added successfully:
Start the task and observe the log
Start two executor instances below and observe the execution of each instance
First configure the local priority configuration of media-service in nacos:
#配置本地优先
spring:
cloud:
config:
override-none: trueStart two instances of media-service
Note that the ports of the two instances cannot conflict when starting:
Instance 1 Add at VM options: -Dserver.port=63051 -Dxxl.job.executor.port=9998
Instance 2 Add at VM options: -Dserver.port=63050 -Dxxl.job.executor.port=9999
For example:
Start two instances
Observe the task scheduling center, wait for a while, there are two executors
Observe the logs of the two execution instances:
The log of another instance is as follows:
From the log, we can see that the fragment sequence number of each instance is different.
If one of the executors hangs up, only one executor is left to work. After a while, the call center finds that there is one less executor and will dynamically adjust the total number of shards to 1.
At this point, the job sharding task debugging is completed