Haven't voted? Don't say you have been to OSC
ElasticJob founder Zhang Liang released the follow-up design plan of ElasticJob on November 4.
ElasticJob is a distributed scheduling solution. In a distributed environment, scheduling directly determines the input and output of running the cluster. The two core elements of scheduling are resource management and trigger timing.
ElasticJob is composed of two independent subprojects ElasticJob Lite and ElasticJob Cloud. ElasticJob Lite is positioned as a lightweight decentralized solution that uses the form of jar to provide coordination services for distributed tasks; ElasticJob Cloud uses a self-developed Mesos Framework solution to provide additional functions such as resource management, application distribution, and process isolation. The following is the planning information just released.
Product Positioning
ElasticJob is currently a fragmented scheduling middleware based on timing tasks, which adds resource management capabilities to ElasticJob-Cloud. In the future, ElasticJob hopes to divide the function into three independent parts: task triggering, resource management, and task governance.
Task trigger is a required module, and the other two modules are optional. Only task triggering is equivalent to downgrading to QuartZ.
Task trigger + task governance can be understood as the status quo of ElasticJob. While task triggering, the ability to increase distributed governance and task fragmentation, future task scheduling based on directed acyclic graphs also belongs to the task management module.
Task trigger + resource management can be understood as a task scheduling mechanism similar to the operating system. Increase resource management and control during task execution, queue tasks when resources are insufficient, and resource management and control can be connected to Kubernetes and Apache Mesos. The current ElasticJob-Cloud is a task trigger + Mesos resource governance implementation method.
Task triggering + task management + resource management are all the capabilities of ElasticJob in the future, while ElasticJob-Lite and ElasticJob-Cloud are only different in deployment form.
Architecture design
ElasticJob hopes to adopt a pluggable architecture design and dynamically weave functional modules into the existing architecture system through SPI. For the three functional modules currently designed, the pluggable designs are:
Task trigger : Currently, it is based on two trigger forms of CRON expression and one-time scheduling. Hope to adjust to scheduling SPI + scheduling implementation (CRON, One-Off, other...).
Resource governance : There is currently no independent abstraction layer for resource governance. I hope that in the future, while increasing the resource management SPI, we will add resource management implementation modules based on Apache Mesos, Kubernetes and NoDep.
Task governance : Currently there are only two parts: task fragmentation and high-availability governance. Hope to provide task management SPI in the future, and use fragmentation and high availability as its implementation modules, and increase DAG management capabilities on this basis. The various modules of task management are isolated from each other and can be superimposed.
ElasticJob-Lite and ElasticJob-Cloud adjustment
ElasticJob-Lite and ElasticJob-Cloud are adjusted to differ only in deployment form.
ElasticJob-Lite adopts a decentralized jar deployment form, provides an access layer for frameworks such as spring, and is suitable for lightweight applications.
ElasticJob-Cloud adopts a centralized server deployment form and is suitable for a centralized job cloud management platform.
Module planning
Task trigger :
Trigger API
Trigger SPI
Trigger Kernel
CRON Trigger
One-Off Trigger
Customized Trigger
Resource governance :
Resource Management API
Resource Management SPI
Resource Management Kernel
NoDep Resource Management
Mesos Resource Management
Kubernetes Resource Management
Task governance :
Job Governance API
Job Governance SPI
Job Governance Kernel
Sharding Job
HA job
DAY Job
Product form :
ElasticJob-Lite architecture adjustment
ElasticJob-Cloud architecture adjustment
ElasticJob-Cloud spins off Mesos
Extended reading: ElasticJob community restarts operations and can schedule 10w concurrent tasks at the same time