[Distributed Task Scheduling] Implementation Principle of XXL-JOB Scheduling Center's Awareness of Executors on and Offline (3)

1 Introduction

List of XXL-JOB special historical articles:

• XXL-JOB scheduling center cluster deployment configuration (1)
• XXL-JOB executor configuration and creation of timed tasks (2)

In the content of the previous two articles, we have obtained a XXL-JOB cluster and a scheduler that can execute tasks. At the same time, we can refer to this process in actual projects to introduce scheduled tasks.

Next, we can explore the implementation principle of the scheduling center's perception of the actuator's offline and offline, mainly including the following points:

• Actuator registration process
• Executor logout process
• Dispatch Center Probing Process

During the running process, if the scheduling center wants to schedule the executors, it must first obtain the information of the executors before it can initiate a scheduling request based on the information. At the same time, we do not want to cause the program to abnormal.

Therefore, XXL-JOB maintains a registration center in the scheduling center, which is realized through the xxl_job_registry table. Every time the scheduling center initiates a scheduling request, it will use the data in this table for load balancing. Then, it is only necessary to register the active executor information, and remove it from the registration center after the executor is shut down or down, so that the dispatch center can obtain the online and offline perception of the executor.

2. Scheduling relationship

How is the scheduled task called?
Let's first look at a hierarchical structure diagram. The scheduling relationship of XXL-JOB is divided into three layers. Each layer is scheduled downward. The top layer is the scheduling center, and the bottom layer is the method that needs to be executed for the scheduled task. The scheduling center can schedule different executions. The executor then invokes its own scheduled tasks, as shown in the following figure:

When the scheduling center schedules the executor, it needs to know the ip and port number of the executor, so as to find the corresponding executor node for scheduling. There are two ways for the dispatch center to obtain the IP of the actuator, namely: automatic registration and manual entry.

Generally, manual entry is not used, why? As you can imagine, when the executor instance is added or reduced, or the executor is down, the machine address needs to be manually modified, which means that someone needs to watch it 24 hours a day, which is a terrible thing.

Therefore, under normal circumstances, we will choose to use the automatic registration method to create. If we choose this method, we need to establish a communication mechanism between the dispatch center and the actuator, and request the transmission of registration information through the network.
Note: The following is the configuration in the background management system.

3. Actuator Registration

The current version (2.3.1) of XXL-JOB uses Http communication, and the dispatch center is realized through SpringBoot.
In fact, the dispatch center starts a Tomcat and provides an executor registration interface. The executor will call this interface when it is started, and transmit its own ip, port and other information to the dispatch center, and then the dispatch center will store it in the database, thus completing the registration of the executor.

3.1. The dispatch center processes the registration request

First, the location of the registration interface exposed by the dispatch center is required.
The naming in the XXL-JOB project is relatively standardized. We can search in the contoller package of xxl-job-admin, and it is easy to find an api-related Controller interface JobApiController and enter this class.
Sure enough, there is an api-related method in this class, as follows:

This is a RESTFUL interface, which includes three policy paths, namely: callback, registry, registryRemove. Through semantics, you can boldly guess that the path of registry is the registration operation, registryRemove is the logout operation, and callback is the execution result of the executor. Callback (this article does not focus on the callback interface for the time being).
We followed the registry path all the way, and found a way to actually do the registration action:

The red box in the figure shows a simple saveOrUpdate method, but here it is done asynchronously, which is easy to understand:

• getXxlJobRegistryDao(): means to get the dao object corresponding to the xxl_job_registry table.
• registrySave: Create an executor registration data.
• registryUpdate: Update the executor information, this operation is used to maintain the heartbeat connection.

Simply put, the dispatch center will receive the registry request from the actuator, and then save the parameters passed in the request into the xxl_job_registy table. This is the executor registration main process run by the dispatch center, a very simple CRUD.

After reading the main process, let's take a look at the details. We can find that the registration code here is not executed synchronously, but asynchronously through a thread pool registryOrRemoveThreadPool. This also reflects a design idea of ​​XXL-JOB, that is, fully asynchronous calls. When we study the follow-up principles, we will often see such usage.

The creation of registryOrRemoveThreadPool
registryOrRemoveThreadPool This thread pool is created in advance when the project is started. It can be found through the usages of Idea. Select registryOrRemoveThreadPool in the code and use the shortcut key alt+F7 to open the interface shown in the figure below and find a new ThreadPoolExecutor() where this creates the thread pool location.

Click the content in the red box to jump to the line of code that created the thread pool to view the thread pool information.

Here you can further view the configuration initialization process of XXL-JOB, and use alt+left mouse button to view the use position of the start() method.

You can see that various initialization operations have been done here. If you want to know a certain process in XXL-JOB later, you can use the initialization operation here as a clue to find the corresponding code flow. In the subsequent source code exploration, Will be in this location many more times.

3.2. The executor initiates a registration request

The so-called executor is actually a Spring-Web project that introduces the xxl-job-core package. In the content of the previous article, we only wrote a @XxlJob annotation in the code to complete a scheduled task method. It is because most of the work is done by the xxl-job-core package, now we can explore how the executor is registered to the dispatch center.

In the registration interface in xxl-job-admin above, an AdminBiz interface is used in this interface, enter this interface, and find the registry method, which has two implementations:

• One is in xxl-job-admin, which is the method of registering in the scheduling center mentioned above.
• The other is in xxl-job-core, which is where the executor initiates the registration request.
  

We can push back through the registration request and find an ExecutorRegistryThread class:

The content in the red box in the above figure shows that a request parameter is composed of appName and address, and then this parameter is transmitted to xxl-job-admin, which is the entry point for actuator registration. Here you can pay attention to while(!toStop), indicating that the current registryThread thread will call the registration method cyclically. Remember the registryUpdate above?
We say that this is used to maintain the heartbeat connection, so how often is the heartbeat request sent? You can pull down the code:

Through the architecture of XXL-JOB, we already know that after the executor is started, the scheduling center is required to do task scheduling, and the scheduling center needs to know the executor's identity, IP address, and port before it can send a scheduling request to the specified executor. This is why there are appName and address in the request parameters in the above figure.

Since the executor has handed over the address to the dispatch center, it is natural to imagine that before handing over the address, the executor will start a web service for the dispatch center to call according to this address.

Continue to jump to the outer layer, you can find the startup code of the web service, here is netty.

In summary, the main process on the actuator side can be represented by a simple diagram:

4. Actuator Logout

The logout of the executor is divided into two types: active logout and passive logout.

• Active logout: As the name suggests, the actuator sends a logout request to the dispatch center, and the dispatch center deletes the registration information of the actuator after receiving it.
• Passive logout: After the executor is down, it cannot normally send a logout request to the dispatch center. The detection thread of the dispatch center finds that an executor has gone offline. At this time, the registration information of the executor is deleted.

4.1. Active logout

The time to initiate active logout is when the Spring container is shut down normally, and the executor class XxlJobSpringExecutor of XXL-JOB implements the DisposableBean interface, which provides a destroy method.

In the subsequent process, the Netty service will be stopped, the detection thread will be interrupted, and a removeRegistry request will be sent to the dispatch center.

After the state of stop is modified, the detection cycle here will stop, and then the following registryRemove method will be called.
After the scheduling center receives the request, it will also perform asynchronous processing through the registryOrRemoveThreadPool thread pool, and finally delete the corresponding executor information in the xxl_job_registry.

4.2. Passive logout

When the scheduling center is initialized, a monitoring thread registryMonitorThread will be started. This thread will trigger a probing operation every 30 seconds (that is, sleep for 30 seconds per cycle). When the probing operation is triggered, it will query the data in the xxl_job_registry table and compare update_time with the current Query the data whose time difference is greater than 90s, and delete this part of the data.

If the sleep time difference is also taken into account, that is, if the executor does not send a new registration request to maintain the heartbeat within 120 seconds at most, the executor will be canceled by the dispatch center.
How is the heartbeat maintained?
After reading the registration process initiated by the executor above, you can probably guess that the registryThread in the executor will call the registration interface of the dispatch center every 30 seconds, and the dispatch center will update the value of update-time after receiving the request.

5. Flowchart

After the above exploration, we have learned about the registration and cancellation process of the executor. The following is the flow chart of the whole process.

6. Summary

The content of this article is mainly to explore the process of registering the executor to the dispatch center and the code implementation. The process is as follows:

1. The dispatch center starts a Tomcat as a web container, exposing the interface of registration and cancellation, which can be called by the executor.
2. After the executor starts the Netty service and exposes the scheduling interface, it transmits its name, ip, and port information to the scheduling center through the registration port of the scheduling center, and calls the registration interface every 30 seconds to update the registration information.
3. Similarly, when the executor stops, it will also request the logout interface of the dispatch center to log out.
4. After the dispatch center receives the registration or cancellation request, it will operate the xxl_job_registry table to add or delete the registration information of the executor
5. The dispatch center will start a probing thread to delete the executors whose registration information has not been updated within 90 seconds.

Since this article is only exploring the process of registration and discovery, the logic related to task scheduling and callback involved in this process is ignored. This part of the logic will be discussed in the next article on the principle analysis of the scheduling process.