About the Author:
Zhang Liang, Dangdang architect, member of Dangdang technical committee, and head of the message middleware group. Strong interest in architecture design, distributed, elegant code and other fields. Currently leading the research and development of Dangdang application framework ddframe, and responsible for promoting and writing technical white papers.
1. Why do you need homework (timed tasks)?
A job is a scheduled task. In general, the system can use message passing in place of some scenarios that use jobs. The two do have similarities. Interchangeable scenarios, such as queue lists. Put the data to be processed into the queue table, and then use a very infrequent scheduled task to pull the data in the queue table and process it. In this case, using the push mode of message middleware can better handle real-time data. And the throughput of database-based message storage is much less than that of file-based sequential append message storage.
But in some cases they are not interchangeable:
Time-driven OR event-driven: Internal systems can generally be driven by events, but when external systems are involved, only time-driven can be used. Such as: grab the external system price. Hourly crawl, since it is an external system, it cannot send event-triggered events like internal systems.
Batch processing OR processing one by one: batch processing of accumulated data is more efficient, and it has advantages over message middleware when real-time performance is not required. And some business logic can only be processed in batches, such as: e-commerce companies settle with express companies, settle once a month, and have commissions based on the number of deliveries. For example, if the monthly delivery exceeds 1,000, an additional 1% discount will be given to the courier company.
Non-real-time OR real-time: Although the message middleware can process data in real time, in some cases, such real-time is not required. For example: VIP users are downgraded, if there is no purchase behavior for more than 1 year, they will be downgraded automatically. This kind of demand does not have a strong time requirement, and there is no need to downgrade VIP users precisely according to the time.
Internal OR system decoupling. Jobs are generally encapsulated within the system, and message middleware can be used for decoupling between systems.
2. What operating system was Dangdang using before?
The operating systems that Dangdang used before were relatively scattered, each fighting for itself, which can be roughly divided into the following 4 types:
Quartz: Java's de facto standard for timed tasks. But Quartz focuses on timed tasks rather than data, and there is no set of processes customized for data processing. Although Quartz can achieve high availability of jobs based on the database, it lacks the function of distributed parallel execution of jobs.
TBSchedule: Ali's early open source distributed task scheduling system. The code is slightly old and uses a timer instead of a thread pool to perform task scheduling. It is well known that timers are flawed in handling exception conditions. Moreover, the TBSchedule job type is relatively simple, and it can only be a mode of obtaining/processing data. There is a serious lack of documentation.
Crontab: Linux system-level scheduled task executor. Lack of distributed and centralized management capabilities.
Perl: Used by legacy systems, it is no longer in line with the company's Java strategy.
3. The origin of elastic-job
elastic-job was originally a part of Dangdang Java application framework ddframe, whose real name is dd-job.
ddframe includes coding specifications, development frameworks, technical specifications, monitoring and distributed components. The ddframe planning is divided into 4 stages of evolution and is currently in stage 2. The technical components involved in stages 3 and 4 do not mean that Dangdang has not been used, but that ddframe has not been uniformly planned.
ddframe consists of various modules, all of which start with dd-, such as dd-container, dd-soa, dd-rdb, dd-job, etc. Dangdang hopes to decouple the various modules of ddframe from the company environment and open source to feedback to the community. DubboX, the previously open source extension of Dubbo, is the core module of dd-soa. The elastic-job introduced this time is the open-source part of dd-job, in which the monitoring (but the monitoring method is open-sourced) and the ddframe core access are not open-sourced.
Fourth, the functions contained in elastic-job
Distributed: The most important function, if the task cannot be executed in a distributed environment, then you can use Quartz directly.
Task sharding: It is the most important and difficult to understand concept in elastic-job. For distributed execution of tasks, a task needs to be divided into n independent task items, and then a distributed server executes one or several fragmented items respectively.
Elastic expansion and contraction: After the task is divided into n task items, each server executes its assigned task items. Once a new server joins the cluster, or an existing server goes offline, elastic-job will trigger task resharding before the next task starts while keeping the execution of the current task unchanged. Example: There are 3 servers divided into 10 slices. Then the shard items are allocated as follows: {server1: [0,1,2], server2: [3,4,5], server3: [6,7,8,9]}. If a server crashes, the shard items are assigned as follows: {server1: [0,1,2,3,4], server2: [5,6,7,8,9]}. If a new server is added, the shard items are assigned as follows: {server1: [0,1], server2: [2,3] , server3: [4,5,6] , server4: [7,8,9] }.
Stability: If there is no server fluctuation, it will not be re-sharded; even if the server fluctuates, the result of the next sharding will calculate a stable sharding order based on the server IP and the hash value of the job name. big change.
High performance: elastic-job will update the necessary information of the job running status to the registry, but in order to consider performance issues, some functions can be sacrificed in exchange for performance improvements.
Idempotency: elastic-job can sacrifice some performance to ensure that the same shard item does not run on two servers at the same time.
Failover: The elastic capacity expansion and contraction will be resharded before the next job runs, but during the execution of this job, the jobs assigned by the offline server will not be reassigned. The failover function can use an idle server to grab orphan job fragments for execution in this job run. The same failover function will sacrifice some performance.
Status monitoring: Monitor the running status of jobs, including data processing functions and the number of failures, job running time, etc. It is idempotent, a necessary function for failover.
Multi-job mode: Jobs can be divided into two modes: simple and data stream processing. Data streams are further divided into high-throughput processing mode and sequential processing mode. High-throughput processing mode can open enough threads to process data quickly, while sequential processing mode The property processing mode assigns each shard item to an independent thread to ensure the order of the same shard, which is similar to the partition order of Kafka.
Some other functions, such as missed task re-execution, single-machine parallel processing, fault-tolerant processing, Spring namespace support, operation and maintenance platform, etc.
5. Deployment and use of elastic-job
Just connect the jar/war of the elastic-job framework to the same Zookeeper-based registry.
The job framework execution data is not limited to the database, and the job framework itself does not associate data. Jobs can be used to process anything like data, files, APIs, etc.
What you need to pay attention to when using elastic-job is to match and process the business processing logic and the fragmentation item allocated by the framework. For example, if the fragmentation item is 1, get the data processing whose id ends with 1. So if you are dealing with data, the best practice is to match the job fragmentation item rules with the data middle layer rules.
As can be seen from the above deployment diagram, job sharding is just a logical concept, and the sharding and actual data actually do not have any matching relationship between the framework. The key to successful use of elastic-job is how the sharded item is related to the actual business. In order not to make the code boring to write, it looks like if(shardingItem == 1) {do xxx} else if (shardingItem == 2) {do xxx}, elastic-job provides custom parameters to shard items The serial number is mapped with the actual business. For example, set it to 1=Beijing, 2=Shanghai. Then the code can use the enumeration of Beijing or Shanghai to fetch data from the Beijing warehouse or Shanghai warehouse in the business. Elastic-job is more concerned with job scheduling and distributed allocation, and it is better to hand over the data to the data middle layer.
As mentioned earlier, the best practice is to match the job sharding item rules with the data middle layer rules, and then adapt the sharding logic of the data middle layer again when job sharding is omitted.
Sixth, the development concept of open source products
In order to reassure those who are interested, I would like to share our development philosophy for open source products.
Code with heart. Code is the only core and output of the project, and any line of code needs to be carefully considered for elegance, readability, and rationality. Elegance seems to be a simple word, but it is actually very difficult to achieve. Everyone has their own understanding of the code in their hearts, and neither elastic-job nor ddframe comes from the hands of one person. The trade-off of code elegance is more difficult to control. The latter items can be understood as supplements to the first item, or specific implementation ideas.
The code is neat and clean to the extreme. To put it simply, it is a severe code cleanliness patient. As long as the code is beautiful and clean, other open source enthusiasts are willing to read the code to find bugs in the project and contribute high-quality code.
Minimalist code, highly reused, no duplicate code and configuration. The Java ecosystem is characterized by an abundance of high-quality open source products. We try our best to consider reusing wheels. For example, lombok is widely used in the project to simplify the code; but we will not use open source products unprincipled. We tend to divide open source products into building blocks and buildings. In the project, we generally only consider using building blocks to build our own buildings, instead of directly using other established buildings. There are two different voices in java-based companies, embracing open source, or not using open source at all. Our view is that since we choose to use java, we should follow the concept of java and embrace the mature things that java has accumulated over the years. Compared with other emerging languages, java may have no advantages in syntax, but there are few other ecosystems comparable in breadth.
A single requirement does not need to consider scalability; two similar requirements are refined. In order not to blindly pursue the so-called ultimate, we use this rule to try to improve the speed of delivery.
The module abstraction is divided reasonably. This is also difficult to measure by standard. Take elastic-job as an example: the core code of elastic-job is divided into 4 blocks, core, spring, console and example; they are used to place core, spring support, console and code example respectively. Do splits at the project level. In core, packages are divided into api, exception, plugin and internal. Used to place externally published interfaces, exceptions, extensible plug-ins provided to end users, and encapsulated internal implementations. Any changes to the internal implementation will not affect the changes of the external interface, and user-defined plug-ins will not affect the stability of the internal code.
If there is no special reason, the test needs to be fully covered. The test coverage of elastic-job core modules is above 95%. Although unit test coverage is not very convincing in a distributed and complex environment, it at least proves that low-level logic errors rarely occur in the project.
Definition of quality. Code Readability > Code Testability > Module Decoupling Design > Functional Correctness > Performance > Functional Extensibility. A project is sustainable only if the code is readable, testable, and 100% in control. Defects can be fixed, performance can be optimized, and the code is unclear, and the project will gradually become a black box. So for frame products, we think quality > time > cost.
Documentation is clear.
7. Future Outlook
The monitoring system needs to be improved. At present, only simple survival and data backlog monitoring can be done through the registration center. The monitoring parts that need to be done in the future are:
1. Add monitorable dimensions, such as job running time.
2. Internal status monitoring based on JMX.
3. Based on historical full data monitoring, all monitoring data are sent to external monitoring centers through flume and other forms to provide real-time analysis functions.
Add task workflow, such as task dependencies, initialization tasks, cleanup tasks, etc.
Real-time improvement of failover function.
More job type support, such as file, MQ and other types of job support.
More sharding strategy support.
The open source address of the project: https://github.com/dangdangdotcom/elastic-job I hope you will pay more attention and contribute code together.
Q&A
Q1: How to judge failure during failover? What are the constraints on the implementation of the task itself?
The failover is currently judged by Zookeeper listening to the temporary node of the shard item. The elastic-job will only be aware that the task hangs after the expiration time of the registry session. There are two forms of failover: 1. The task hangs up, and elastic-job will find an idle job server (may be an unassigned task, or it may complete the execution of this task) to execute. 2. If there is no idle server at that time, the unassigned shard item will be grabbed when a server completes the assigned task.
Q2: Is the role of Zookeeper to save task information? Will it affect task execution if Zookeeper hangs?
Zookeeper's current znodes are divided into four categories, config, servers, execution, and leader. config is used to save global control of distributed jobs, such as how many shards are divided, whether to execute misfire, and cron expressions. servers is used to register job server status and shard information. execution stores job runtime state in the dimension of shards. The leader is used to store the master node. The elastic-job job execution is decentralized, but the master node plays a coordinating role, such as resharding, cleaning up the last runtime information, etc.
Q3: Can it be integrated with spring batch in task processing?
Spring batch has paid attention to it before, but currently elastic-job has not been integrated. The spring support of elastic-job customizes the job namespace, simplifies spring-based configuration, and can use spring-injected beans. Spring batch is also a good job framework, including spring-quartz, but the distributed function is not mature. Therefore, it is more difficult to change on top of this, and elastic-job prefers to make a green product that does not depend on spring, but can be integrated into spring.
Q4: For simplicity and data flow, can you talk about how specific shards are handled?
A simple job is not encapsulated by any business logic, but only provides an execute method, which is triggered regularly, but adds a distributed sharding function. It can be simply understood as the distributed version of quartz. Although quartz can support database-based distributed high availability, it cannot be sharded. That is to say, two servers can only have one master and one backup, and cannot run load balancing at the same time. Data flow type jobs refer to Alibaba's previous open source TBSchedule, which divides data processing into fetchData and processData. The data is first taken out from the database, file system, or other data sources, and then processData is centrally processed, which can be processed one by one, and can be processed in batches (this will be added in the future). processData is executed by multiple threads, and data flow type jobs can be subdivided into two types, one is high throughput and the other is sequential. High throughput can open any number of threads to perform data processing in parallel, and sequential execution will use one thread per shard item to ensure the order of data in shard items, which refers to the implementation of kafka. The data stream type job has the parameter isStreaming, which is used to control whether to process the data in a non-stop stream, similar to a perpetual motion machine, as long as there is data, it will be processed all the time. But this kind of job is not suitable for scenarios where every fetchData puts a lot of pressure on the database.
Q5: How to realize that a task is only executed once on one node?
The current idempotency is to add the registration of the state of the shard item in the znode of execution. If the state is running, even if another server wants to run the shard item, elastic-job will refuse to run, instead Wait for this state to change to a non-running state. The state is updated when each job shard item starts. When the server does not fluctuate, there is no situation where a shard is divided into two servers. But once the server fluctuates, this may happen at the moment of sharding. Regarding sharding, it is actually a more complicated implementation. At present, sharding is to detect server fluctuations, or modify the total number of shards, and will record a status instead of direct sharding. The sharding will be executed when the next job is triggered, only the master node can shard, and the slave nodes in the shard will be blocked. One of the biggest problems with non-scheduled distributed jobs is that there is no guarantee that the master node job must be triggered before other slave nodes. Therefore, it is very likely that the slave node will trigger the execution first and use the old shard; then the master node will re-shard, which will cause the job shard to be inconsistent this time. This requires execution nodes to ensure idempotency. The next time it is executed, as long as there is no server fluctuation, the previously wrong shards will naturally be corrected.
Q6: If Zookeeper hangs, are all tasks suspended and cannot be run, including those that have been run once, if they resume, can the tasks run normally, or should the business application service be restarted?
In fact, Zookeeper is not easy to hang. After all, Zookeeper is distributed and highly available, generally not a single one. At present, the fault tolerance of elastic-job is that the job server that cannot connect to Zookeeper will stop executing the job immediately, preventing the master node from being re-sharded, and the split-brain server is still executing. That is, Zookeeper hangs and all jobs will stop. And once the job server is back connected to Zookeeper, the job will resume running. Therefore, if Zookeeper hangs, the data will not be affected, and if Zookeeper is restored, the job will continue to run without restarting.
Q7: Can it be specific to the business level? For example, there is a task, which is to send 100w user emails. How should we split it at this time? How is paging for distributed databases handled here?
For emails of 100W users, I personally think that we can take the modulo according to the user id. For example, it is divided into 100 shards, the whole userid is % 100, and then each shard sends an email with the userid ending in the modulo result. In detail: shard 1 sends emails with userid ending in 01, ..., shard 99 sends emails with userid ending in 99. The paging of distributed databases, theoretically speaking, is not the scope of job framework processing, and should be handled by the data middle layer. By the way, the data middle layer part of ddframe, sharding-JDBC will be open source early next year. By modifying the JDBC driver to implement sub-database sub-table. It is not a middleware method such as MyCat or cobar; nor is it based on an ORM method such as hibernate or mybatis. sharding-JDBC is relatively lightweight and easier to adapt to various databases and ORMs
Q8: Is ddframe composed of many components? Does it support multiple languages?
ddframe is an umbrella term for many components. It is divided into core modules, distributed component modules and monitoring docking modules. The core module can be understood as spring-boot, something that can quickly start and build projects quickly.
Distributed components include Dubbox called by SOA, elastic-job based on distributed jobs, and sharding-JDBC just mentioned, as well as modules such as cache, MQ, and NoSQL that have no open source plans in the near future.
It is estimated that the monitoring module will not be open sourced in the future, and it is too tightly bound to the company's own business scenarios. It is mainly divided into log center, traffic analysis and system relationship call graph. The monitoring part is still being done, and it is not very powerful.
In terms of multi-language, SOA module support, Dubbox's REST extension is to support calls in other languages. The rest are temporarily unavailable. Such as sharing-JDBC, mainly based on java JDBC, if there are multiple languages, the middle layer is a better method.
The module names of ddframe are dd-*, dd-soa, dd-rdb, dd-job, dd-log and so on. elastic-job, sharding-JDBC, etc., are names that were extracted and re-created from ddframe for open source.
https://www.cnblogs.com/qiumingcheng/p/5573845.html