Distributed Task Scheduling: What You Know and Don't Know

Introduction

Everyone should be familiar with timed tasks. From the ashes-level Crontab to Spring Task, from QuartZ to xxl-job, as business scenarios become more and more complex, the timed task framework is constantly being upgraded and evolved.

So today, I will talk with you about distributed task scheduling from the following three aspects: the evolution process from stand-alone timed tasks to distributed task scheduling platforms, how Tencent Cloud distributed task scheduling platform TCT came into being, and the specifics of TCT. The situation of the landing case and the core problems solved.

about the author

Cui Kai

Tencent Cloud CSIG Microservice Product Center Product Architect With many years of experience in R&D and system architecture design of distributed and high-concurrency e-commerce systems, he is good at the implementation and implementation of mainstream microservice architecture technology platforms. Currently, he focuses on the research and promotion of microservice architecture-related middleware. And the precipitation of best practices, dedicated to helping enterprises complete digital transformation

scene type

There are thousands of scenarios of timed tasks, but what is its essential model? How to understand timed tasks? Here is a comparison. Timed tasks are actually the teachers assigning homework to students , such as:

每天晚上7点准时开始写作业，写完了让家长检查签字。

"Punctuality at 7 o'clock every night" is a requirement for time accuracy and periodicity, 7 o'clock not 8 o'clock, every day not every week; "homework" is to clarify the content of the task execution, it is homework not watching Ultraman ; "Let parents check and sign after writing" makes "homework" and "parents check and sign" decoupled into two logical actions, and parents can read books or something while their children are writing homework.

Closer to home, the typical implementation scenarios of timed tasks are very common in various industries: in e-commerce, the entrance of promotions is opened regularly, the receipt is automatically confirmed if the receipt is not confirmed within 15 days, and the unpaid orders are scanned at fixed points for SMS reminders, etc.; in the financial and insurance industry, There are also scenarios such as commission calculation for marketers, production of terminal marketing reports, timing synchronization of organizational relationships, and daily settlement and monthly settlement. To sum up, the author 时间驱动、批量处理、异步解耦divides the scene types of timed tasks according to three dimensions.

time driven

Taking the timed opening of the event entry in the e-commerce scenario as an example, in general, various parameters required for the event are configured in the background, and the dynamic configuration of the event status is set to off. When the execution time is reached, the scheduled task is automatically triggered and the promotion is started. .

It can be seen that in a time-driven scenario, compared to the execution content, the business is more concerned with time factors such as whether the task is executed regularly or periodically, the accuracy of the specific time point of execution, and the length of the cycle frequency.

batch processing

Batch processing tasks are characterized by the need to 同时对大量accumulate business objects for processing. At this point, some friends may ask, why not use message queue processing? The reason is that in some specific scenarios, message queues cannot be simply replaced, because message queues are more event-driven through each message, and tend to be more real-time processing.

Description of the settlement business based on commissions in insurance, such as the calculation of commissions for marketers. Marketers will get a certain percentage of commission from the premiums paid by the insured, and this percentage will vary according to the insurance period and different types of insurance. In addition, some commission incentive policies of the company may be superimposed. In such a scenario, it is necessary to accumulate a certain amount of data, and then perform batch calculations at regular intervals, instead of calculating each event.

Asynchronous decoupling

When it comes to the asynchronous decoupling of the system, one must think of the message queue, but the message queue is not suitable for the acquisition of some external system data, such as the capture of the exchange stock price by the stock software company in the securities industry. The company is external data, and it is difficult to communicate asynchronously between internal and external systems using message queues. Therefore, in general, the data will be captured and stored regularly through batch tasks, and then the back-end system will analyze and organize the data, so that the external data acquisition and the internal data analysis and processing are logically decoupled.

Past and Present

Stand-alone scheduled tasks

Single-machine timed tasks are the most common and more traditional task execution methods, such as the built-in Crontab of Linux. It implements the execution of single-machine scheduled tasks through the time dimensions of minutes, hours, days, months, and Fridays in cron expressions.

# 每晚的21:30重启smb
30 21 * * * /etc/init.d/smb restart

In addition, there are also built-in timing tasks in java, such as the java.util.Timer class and its upgraded version of ScheduledThreadPoolExecutor. In addition, Spring Task is also provided in the Spring system, which is a single-machine timing task framework that supports cron expressions quickly through annotations. .

@EnableScheduling
@Service
public class ScheduledConsumerDemo {

    @Value("${consumer.request.echoUrl}")
    private String echoUrl;

    /**
     * 间隔1秒请求provider的echo接口
     *
     * @throws InterruptedException
     */
    @Scheduled(fixedDelayString = "${consumer.auto.test.interval:1000}")
    public void callProviderPer1Sec() throws InterruptedException {
        String response = restTemplate.getForObject(echoUrl, String.class);
    }
}

Obviously, single-machine scheduled tasks are very convenient to deal with simple business scenarios, but now that the distributed architecture has become a trend, single-machine scheduled tasks cannot meet the demands of enterprise-level production and industrialized scenarios , which are mainly reflected in the unified configuration of cluster tasks. Management, single point of failure and single point performance, communication and coordination of tasks between nodes, task execution data aggregation, etc. In order to meet the demands of enterprise-level production, various task scheduling platforms are gradually emerging.

Centralized scheduling

The typical centralized scheduling framework quartz, as the predecessor and leading brother of the task scheduling industry, once became synonymous with task scheduling through its excellent scheduling capabilities, rich API interfaces, and good Spring integration.

In the quartz architecture, database locks are used to ensure the uniqueness of multi-node task execution, which solves the problem of single point of failure. However, the centralization of database locks also causes serious performance problems. For example, in the scenario of large-scale tasks, the database becomes the performance bottleneck of the overall business scheduling. At the same time, it will cause some resources to wait idle on the application side, and the task cannot be completed. of parallel sharding .

Another framework xxl-job from Dianping is mainly characterized by simplicity, easy integration, and visual console. Compared with quartz, the main differences are:

• Self-developed scheduling module :

The asynchronous design of xxl-job decouples the scheduling module and the task module , which solves the problem that the performance of the scheduling system is greatly reduced when the scheduling task logic is heavy. Among them, the scheduling module is mainly responsible for the analysis of task parameters and initiating calls, and the task module is responsible for the execution of task content. At the same time , the optimization of the asynchronous scheduling queue and the asynchronous execution queue enables limited thread resources to support a certain amount of job concurrency.

• Scheduling optimization :

By scheduling thread pools and parallel scheduling , the probability of scheduling blocking is greatly reduced, and the load capacity of the scheduling system is increased at the same time.

• High Availability Guarantee :

The database of the scheduling center will save task information, scheduling history, scheduling logs, node registration information, etc., and ensure data persistence and high availability through MySQL. The failover mode and heartbeat detection of task nodes also dynamically senses the status of each execution node.

However, since xxl-job uses a database lock mechanism similar to quartz, it is also unavoidable that the database becomes a performance bottleneck and other problems caused by centralization.

Decentralized scheduling

In order to solve various problems in centralized scheduling, the domestic open source frameworks are also the eight immortals crossing the sea and showing their magical powers, such as powerjob with good reputation, elastic-job of Dangdang, and saturn of Vipshop. As a whole, saturn is improved and optimized based on the open source elastic-job, so this article only briefly introduces powerjob and elastic-job.

Powerjob was born in April 2020, which contains some relatively new ideas and elements, such as support for MapReduce-based distributed computing, dynamic hot loading of Spring containers, etc. Functionally, multi-task workflow orchestration, MapReduce execution mode, and delayed execution are the highlights. At the same time, it is claimed that all components support horizontal expansion. The core components are described as follows:

• powerjob-server: scheduling center, unified deployment, responsible for task scheduling and management;
• powerjob-worker: executor, providing single-machine execution, broadcast execution and distributed computing;
• powerjob-client: optional component, OpenAPI client.

The lock-free scheduling design idea of ​​powerjob in solving centralized scheduling is worthy of reference. The core logic is to use appName as the key of business application grouping, and logically bind powerjob-server and powerjob-worker with grouping keys , that is, to ensure that each powerjob- The worker cluster will only be connected to one powerjob-server when it is running, so there is no need for a locking mechanism to prevent tasks from being acquired by multiple servers at the same time, resulting in repeated execution problems.

Although powerjob is relatively excellent in all aspects, after all, the product iteration cycle is relatively short, and it is still necessary to continuously polish the product details through large-scale application in the market to verify the performance, ease of use and stability of the product.

elasticjob includes two independent sub-projects, elasticjob-lite and elasticjob-cloud. This article mainly takes elasticjob-lite as an example.

Elasticjob-lite is positioned as a lightweight and decentralized solution. On the basis of inheriting quartz, zookeeper is used as the registration center. At the product design level, I personally understand that elasticjob focuses more on data processing and calculation than other distributed task scheduling frameworks, which are mainly reflected in the following two aspects:

Decentralization of elasticjob-lite :

• There is no scheduling center design. After the jar package of elasticjob is introduced into the business program, the jar package performs tasks such as task scheduling, status communication, and log placement.

• Each task node is peer-to-peer, and will register task-related information (task name, peer instance list, execution strategy, etc.) in zookeeper, and rely on zookeeper's election mechanism to elect execution instances.

Elastic sharding for elasticjob-lite :

• Based on zookeeper, task execution instances can perceive each other's online and offline status in near real time, so that the allocation of task shards can be adjusted as the number of task instances is adjusted, and the load is relatively uniform.
• When a task instance goes offline, it will not affect the current task, and it will be re-sharded when the next task is scheduled to avoid repeated execution of the task.

Through the above analysis, elasticjob is more designed for distributed task computing scenarios, and is more suitable for fragmented computing or processing of large amounts of data, especially in scenarios that require resource utilization.

evolution

After a rough introduction to various mainstream distributed task scheduling frameworks, a question arises: what are the main factors that drive the framework to evolve step by step? The author briefly summarizes the following four factors:

• Business complexity : The original business complexity is low, and it can be done with 2 or 3 lines of code; with the increase of business complexity, the organizational form and execution content of tasks have undergone great changes, and task scheduling and framework ecological integration have gradually been derived. , multi-language and multi-terminal support .

• Diversity of scenarios : It is no longer just simple scheduled task execution. Problems in scenarios such as batch computing and business decoupling are gradually solved by using the distributed task scheduling framework. The requirements for framework capabilities lie in richer task execution strategies, support for dynamic sharding computing, and rich task governance capabilities .

• Distributed Architecture : The full arrival of the distributed architecture trend is the most important driving factor. The overall design of the framework must be based on a distributed architecture . Communication between task nodes and scheduling centers, high availability of the scheduling platform, fault handling and recovery of task nodes, and visual operation and maintenance of task scheduling are all new challenges.

• Massive data concurrency : When massive business data and concurrent calls become the norm, the distributed task scheduling platform needs to be optimized in terms of executor performance, execution time accuracy, parallel and asynchronous processing of tasks, and flexible management and control of node resources . Helps improve the overall throughput of the platform.

The evolution of the distributed task scheduling framework is a branch of the evolution of business systems from a single architecture to a distributed architecture. The continuous improvement of the capabilities of the distributed task scheduling platform is inseparable from the microservice evolution of the business architecture.

In the same way, the business systems of various industries are gradually migrated to the cloud, and the digital transformation trend of enterprises is obvious. The evolution of the distributed task scheduling platform in the future will also be inseparable from the cloud-native industrial environment. The overall architecture of the platform needs to be deeply integrated with the cloud-native system. Meet the changing industrial demands in many aspects in the future.

TCT "on the cloud"

Distributed Task Scheduling Service (Tencent Cloud Task) is a lightweight and highly reliable distributed task scheduling platform independently developed by Tencent Cloud. By specifying time rules, scheduling tasks are strictly triggered to ensure reliable and orderly execution of scheduling tasks. This service supports international time expressions, performs life cycle management, and solves the problems of single point of failure and low visibility of traditional scheduled tasks. At the same time, it supports complex scheduling task processing capabilities such as task sharding and workflow orchestration, covering a wide range of task scheduling application scenarios, such as data backup, log segmentation, operation and maintenance monitoring, and financial daily cutting.

Features

TCT is mainly divided into three parts in terms of functions: scheduling management platform, task scheduling service, and development integration (SDK). The scheduling management platform provides elegant visual interface interaction, the task scheduling service realizes task scheduling in distributed scenarios, and develops and integrates an open source framework for deep integration. The detailed functional characteristics are described as follows.

Rich task configuration

• Multiple execution modes: support random node, broadcast, and fragment execution modes to meet different application scenarios.
• Multiple trigger strategies: support timing trigger, cycle trigger, workflow trigger, manual trigger strategy.
• Perfect fault tolerance mechanism: Supports various task fault tolerance protection mechanisms such as abnormal retry, timeout interruption, and manual stop.

Visual task management

• Task management view: Display the execution status of tasks, and provide operation capabilities such as adding tasks, editing tasks, deleting tasks, manual execution, and starting/deactivating tasks.

• Execution record view: Displays the detailed list of execution batches of all routine tasks and workflow tasks, and supports query filtering conditions based on the tasks and deployment groups to which they belong.

• Execution list view: Displays the detailed list of execution batches of the selected task, and supports stop and re-execution of task batches.

• Execution details view: Displays the list of execution instances of task execution batches, and supports stop, re-execution, and log query operations for execution instances.

• Workflow management view: Displays the execution status of workflow tasks, and provides operational capabilities such as creating workflow tasks, visualizing process orchestration, and enabling/disabling workflow tasks.

Perfect task operation monitoring and alarm

• Three-dimensional monitoring: Provides three-dimensional monitoring of task running status, task execution batch status, and execution instance running status, and supports online log viewing capabilities for execution instances.

• Flexible alarm strategy: The integrated cloud monitoring capability provides task execution batches, execution instance exception alarms, workflow task execution batches, batch tasks, execution instance exception alarm capabilities, and supports flexible indicator alarm and event alarm configuration.

Architecture Principles

The brief introduction of each component of TCT is as follows :

• Trigger: Trigger rules for parsing tasks;
• Scheduler: dispatch tasks that need to be executed, manage task status, etc.;
• Monitoring: report monitoring data related to task execution;
• Console: the administrator's console interface;
• Access layer: the channel manager for messages such as task distribution and status reporting;
• Access gateway: a gateway that connects the access layer and SDK in a unified manner;
• SDK: runs together with the business process and is responsible for executing a specific piece of code logic defined in the task.

First, the task information configured by the user in the console and stored in the DB is parsed by the trigger, and the parsed execution information is put into the MQ. Secondly, the scheduler consumes the execution information and sends it to the specific executor node through the access layer (there is specific node registration information in the access layer, including IP address, etc.). Finally, when the node where the SDK is located completes the execution of the task (success, failure, unresponsiveness, etc.), the execution result will be sent back to the scheduler through the TCP long connection, and then the scheduler will interact with the DB to complete the task status change, and at the same time Report the task execution status to the monitoring module.

Through the function introduction, it can be found that TCT basically covers the functions required in common task scheduling scenarios, especially in the visual view, and has done a lot of work. Great protection is also provided. In addition, TCT originates from the TSF technology platform and integrates TSF applications naturally. The support components can easily obtain relevant information of TSF applications, such as TSF deployment group ID, node IP, application ID, etc., so the task execution efficiency will also be improved. high.

However, it can be found from the overall architecture diagram that TCT adopts a centralized scheduling scheme. The scheduler, trigger and console components are stateless, support horizontal expansion, and communicate between components and SDK through TCP long-term connection; while the data flow depends on DB and MQ. In a large-scale implementation scenario with a large number and high execution frequency, the throughput of DB and MQ will become a performance point, and even if it can be optimized, there will be an obvious upper limit. Therefore, according to the current product form of TCT, it is more suitable for lightweight task scheduling scenarios.

Sharding execution case

Background overview

The sharding execution mode is a frequently used execution method in the scenario of large-scale data processing. This case is illustrated by the business scenario in which the subsidiary in the insurance industry summarizes the marketing data of the day to the head office every day.

As can be seen from the above figure, the service for summarizing marketing data (hereinafter referred to as summarydata) regularly calls the marketing data query API provided by 34 subsidiaries at 2:00 am every day. The sharded execution method is used because the operations for aggregating marketing data need to be triggered at the same time, and the shorter the overall aggregation time, the more accurate it is. In addition, the amount of marketing data varies from company to company, and even the same subsidiary produces different amounts of marketing data on a daily basis.

Configuration steps

According to the above business background description, and based on the existing resources, the overall configuration ideas are as follows:

• Create a summarydata deployment group, in which 4 new instances are created, and the number of thread pools for a single instance is 3 ;
• In the application code, the 34 subsidiaries are one-to-one corresponding to the company IDs 1~34 ;
• Divide 34 companies into four regions: NORTH, SOUTH, EAST, and WEST according to the approximate region and the average daily data volume;
• The number of shards is 4, and each shard corresponds to 1 instance, that is, 1 instance calculates data of at least 1 area;
• The subsidiary ID list corresponding to each regional key can be semi-automatically adjusted through code configuration to prevent a sudden increase in the data volume of a subsidiary;
• To prevent repetition of statistics, automatic task retry is not configured, and manual compensation is used.

Step 1: Write and package trigger class code

public class SimpleShardExecutableTask implements ExecutableTask {

    private final static Logger LOG = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());

    @Override
    public ProcessResult execute(ExecutableTaskData executableTaskData) {
        // 输出任务执行元数据
        TaskExecuteMeta executeMeta = executableTaskData.getTaskMeta();
        LOG.info("executeMetaJson:{}",executeMeta.toString());
        // 输出分配给本实例的分片参数
        ShardingArgs shardingArgs = executableTaskData.getShardingArgs();
        LOG.info("ShardCount: {}", shardingArgs.getShardCount());
        Integer shardingKey = shardingArgs.getShardKey();
        LOG.info("shardingKey: {}", shardingKey);
        String shardingValue = shardingArgs.getShardValue();
        LOG.info("shardingValue: {}", shardingValue);
        // 模拟任务执行
        try {
            this.doProcess(shardingValue);
        } catch (Exception e) {
            e.printStackTrace();
        }
        return ProcessResult.newSuccessResult();
    }

    public void doProcess(String shardingValue) throws Exception {
        if (shardingValue.equals(CompanyMap.NORTH.area)){
            Arrays.stream(CompanyMap.NORTH.companyIds)
                    .forEach(companyId->LOG.info("calling north subsidiary_{} api.....",companyId));
        } else if(shardingValue.equals(CompanyMap.SOUTH.area)){
            Arrays.stream(CompanyMap.SOUTH.companyIds)
                    .forEach(companyId->LOG.info("calling south subsidiary_{} api.....",companyId));
        } else if(shardingValue.equals(CompanyMap.EAST.area)){
            Arrays.stream(CompanyMap.EAST.companyIds)
                    .forEach(companyId->LOG.info("calling east subsidiary_{} api.....",companyId));
        } else if(shardingValue.equals(CompanyMap.WEST.area)){
            Arrays.stream(CompanyMap.WEST.companyIds)
                    .forEach(companyId->LOG.info("calling west subsidiary_{} api.....",companyId));
        }  else {
            throw new Exception("input shardingValue error!");
        }
        ThreadUtils.waitMs(3000L);
    }

    enum CompanyMap{
        NORTH("NORTH", new int[]{1, 2, 3, 4, 5, 6, 7, 8, 9}),
        SOUTH("SOUTH",new int[]{10,11,12,13,14,15,16,17,18,19}),
        EAST("EAST",new int[]{20,21,22,23,24,25,26,27,28}),
        WEST("WEST",new int[]{29,30,31,32,33,34});

        private String area;
        private int[] companyIds;

        CompanyMap(String key,int[] values){
            this.area = key;
            this.companyIds = values;
        }

        public String getArea() { return area; }
        public void setArea(String area) { this.area = area; }
        public int[] getCompanyIds() { return companyIds; }
        public void setCompanyIds(int[] companyIds) { this.companyIds = companyIds; }
    }
}

Step 2: Create an application and deployment group, and complete the deployment

Step 3: Create a TCT task

Step 4: Manually start the task test

Test effect

View the execution status of the instance through the console, and at the same time, you can use the sharding parameter button to query the sharding parameters in the execution batch of an instance.

Viewing the results through the application log, it can be found that one instance runs two sharding tasks. This is because TCT judges the instance load and selects a relatively idle instance.

In addition, the test of the abnormality of the instances in the service was also carried out, that is, the execution of the task when only 1 of the 4 instances in the summarydata service is normal (due to the long log, the author has selected important parts). It can be seen that the first three sharding tasks are executed at the same time and using different threads, and the fourth sharding task is executed after the first three tasks are executed, which is in line with expectations.

Future direction

The competition process between distributed task scheduling platform frameworks is long and stalemate. Various manufacturers are looking for breakthroughs in product value. There are still many shortcomings in TCT, and it is necessary to continue to think deeply from the perspective of market demand and technology trends. For the distributed task scheduling market, the author roughly summarizes the following possible optimization directions for future products:

1. decentralization

The centralized distributed task scheduling platform has obvious shortcomings, and it is difficult to support the large-scale implementation of enterprises. At the same time, the evolution trend of products and technologies in the market is gradually developing towards decentralization. The reason is that a decentralized distributed task scheduling platform has the possibility of large-scale commercialization. Successful commercialization cases are also a sign of product maturity. .

2. containerized

Distributed task scheduling platform components and inter-component communication are mostly traditional virtual machines. If the containerized deployment of supporting components can be realized at the same time, the advantages of the container platform in terms of rapid start and stop, resource scheduling, and horizontal expansion can be better utilized. , in order to improve the overall availability of the support side, reduce the operation and maintenance costs when expanding and shrinking capacity, and effectively improve the overall throughput of the platform. High availability, elastic scaling, and high performance are important considerations for large-scale enterprises to digitally transform cloud-native.

3. programmable

More and more distributed task scenarios require complex task scheduling for multiple tasks. At present, the mainstream scheduling is still limited to simple logical processing such as serial, parallel, and OR between tasks. In the future, more general and programmable template language is needed to describe task parameters and content, DAGS (directed acyclic graph), operators, trigger actions, etc., and standardize the way manufacturers define task scheduling. .

4. fault compensation

There are also many aspects that need to be improved in the processing strategies when tasks and workflows are executed abnormally, such as outdated triggering problems caused by instance ramming, task chasing and task accumulation problems, and overall retry or breakpoint continuation after task exceptions in workflow scenarios retry, etc.

5. scene upgrade

At present, various products have a relatively high degree of functional homogeneity in common timed task scenarios. However, with the rapid development of cloud native, big data and other related fields, distributed task scheduling platforms have gradually produced new application scenarios, such as distributed computing and computing aggregation in big data scenarios, and scheduling platforms docking with serverless applications. All of them put forward higher requirements for the scene and function of the product.

Epilogue

Through the introduction of timed task scenarios, evolution history, various platform frameworks, and the practical case description of Tencent Cloud's self-developed distributed task scheduling framework TCT, the author continues the application status and future development of distributed task scheduling framework on the basis of predecessors. After a brief analysis, readers and friends who knew and didn't know before should know it. I hope this article can provide some ideas and perspectives in technology selection and open source construction for reference by enterprises and open source communities.

quote

https://cloud.tencent.com/document/product/649/45253 http://www.quartz-scheduler.org/ https://www.xuxueli.com/xxl-job https://shardingsphere.apache.org/elasticjob/current/cn/overview/ http://www.powerjob.tech/ https://vipshop.github.io/Saturn/#/zh-cn/3.x/

{{o.name}}

{{m.name}}