User case | Application and practice of Apache DolphinScheduler offline scheduling in Ziroom multi-service scenarios

file

User Case | Ziroom

With the rapid development of Ziroom's business, the ever-increasing number of scheduling tasks and more than 10,000 tasks in stock have put forward higher requirements for platform stability. At the same time, many non-professional developers also need a more "people-friendly" scheduling platform experience.

How to meet these increasingly prominent demands is undoubtedly a huge challenge for the development team of the Ziroom big data platform. After in-depth research and comparison, the team found that Apache DolphinScheduler is a project that can meet all the current core requirements of Ziroom.

So far, the team started to introduce Apache DolphinScheduler, and carried out a series of transformations and optimizations on this basis, including automatic generation of SQL lineage parsing schedules, support for Ambari to manage Apache DolphinScheduler, and availability monitoring of end-to-end scheduling components. Better meet the needs of the enterprise.

This article will describe in detail the application and evolution of Apache DolphinScheduler in Ziroom.

About the Author

Chen Zhuoyu, Ziroom big data platform operation and maintenance, responsible for Ziroom offline data scheduling, Apache StreamPark PPMC, Apache DolphinScheduler Contributor

business challenge

Complex business scenarios: Ziroom's data processing business scenarios are rich and diverse, covering four major sectors: To C and To B quality residential products, smart living services, smart home improvement, and smart community organization and operation.
A large number of historical stock tasks: Ziroom has accumulated a large number of offline tasks in history. At present, the accumulated offline scheduling tasks have reached a scale of 10,000+, which puts forward very high requirements for the stability of the platform.
Large increase in offline tasks: Currently, Ziroom is still in the stage of rapid business development, and the number of offline tasks continues to grow, which poses extremely high challenges to the scalability and processing capabilities of the platform.
Ease of use requirements of non-professional developers: Free data users mainly include non-professional developers such as operators, analysts, and product BPs. They have strict requirements on the ease of use of configuration scheduling and need to be able to support SQL-based operations, and User-friendly configuration interface to achieve a "civilian" experience.

solution

Ziroom's core demands for scheduling technology selection

Ziroom's selection requirements for scheduling technology can be analyzed from two aspects: one is the user level, and the other is the technical operation and maintenance level.

At the user level, our users expect the platform to provide:

Simple and easy-to-use operation mode: enables users to get started quickly and efficiently develop demand logic.
Rich practical cases: for users' reference and learning, to help them better understand and use the platform.

At the level of technical operation and maintenance, our development and operation and maintenance teams expect the platform to provide:

Universal technology stack: It is convenient for secondary development and quickly integrates projects into the free enterprise ecosystem.
Rich components: Support a variety of task types to meet various business needs.
Excellent architectural design: ensure that the project has high availability, easy scalability, and the ability to support massive task scheduling.
Active open source community: When encountering problems, the team can easily and quickly obtain necessary help from the community.

In response to the above core demands, the Ziroom team conducted in-depth research on all related projects in the industry, and finally found that Apache DolphinScheduler is the only project that can meet all the core demands of the Ziroom team. Therefore, we chose Apache DolphinScheduler2.0.6 version as the free offline scheduling technology solution.

architecture design

At present, Ziroom has successfully built a big data platform that can be used by the whole group through internal research and development. In order to further meet the needs of the vertical field of offline data warehouse, Ziroom chooses to use Apache DolphinScheduler for integration and expansion. This improvement aims to enhance the capabilities of the overall platform, so that it can provide more powerful data processing and data orchestration capabilities for various business departments within the group.

file

As shown in the figure, data scheduling is the fourth link in the Ziroom data development process, and it belongs to the core of the whole process. In the upstream of the process, the lineage function of Hetu (data lineage query service) and the query capability of Hue jointly provide support for generating SQL business logic, while the role of data scheduling is to arrange and process these SQL logics.

In the downstream of the process, the first is the data quality module. The core responsibility of this module is to ensure the accuracy of the scheduling output, which is what we call "caliber" assurance. Through this link, we ensure the quality of the data and provide a reliable basis for the next steps. The second is the data service module. The main task of this module is to summarize and process the scheduling results again to make it meet our expectations at the index level, and provide it in the form of standardized Restful services for users to use. .

Secondary development practice

Lineage analysis automatically generates scheduling

file

As shown in the figure above, the complexity of the SQL task node can be seen from the many upstream dependent table relationships involved, and nearly 30 table relationships are intertwined with each other. In the face of such complex dependencies, if it is only done by business personnel, they not only need to spend a lot of time to understand the relationship between the company's related business line tables, but also need to master the configuration and use of DolphinScheduler. Said to be a heavy burden.

Therefore, the Ziroom team has found a new solution: lineage analysis automatically generates schedules. The core value of this solution is that it completely liberates business personnel from tedious scheduling configurations. Just provide the SQL statement, the program will read the SQL, parse it, and get the input table and output table required for scheduling. The parsed results will be used to automatically generate and configure schedulers.

file

As shown in the figure, the user only needs to synchronize the debugged SQL in Hue to the scheduling platform with one click. Here, just click the "Resolve" button, and the system will automatically resolve all relevant dependencies according to the blood relationship of SQL, and automatically generate a directed acyclic graph (DAG) of the task based on this information. This design aims to simplify the user's operation steps to the greatest extent, while ensuring the accuracy and efficiency of scheduling tasks.

Therefore, the original complex configuration process is simplified to provide SQL statements for the user, and the rest is done by the system. In this way, business personnel only need to focus on business needs and complete work efficiently and concisely, without having to be troubled by complex configurations and technical details in order to meet these needs.

Support mixed data dependencies and task dependencies

file

There are nearly 10,000 scheduling tasks in Ziroom's internal history. These tasks are scattered on multiple scheduling platforms such as self-developed scheduling of business lines and Airflow scheduling. The team does not intend to uniformly migrate these historical stock tasks to the new scheduling platform, but prefers to maintain the status quo while meeting new requirements. Based on this consideration, Ziroom designed and implemented the "data dependency" function on DolphinScheduler, enabling DolphinScheduler to support mixed dependency modes, thus providing better compatibility for scheduling platforms.

In this way, DolphinScheduler can handle both "task dependencies" and "data dependencies". "Task dependency" refers to triggering workflow execution by detecting whether the task is executed successfully, and "data dependency" is to determine whether the workflow is executed downward by detecting whether the data partition is generated. This mixed dependency processing method enables DolphinScheduler to effectively link new tasks and historical scheduling tasks between business lines to form a unified and efficient scheduling system.

Task Dependency Refactoring

DolphinScheduler uses the DEPENDENT plugin to implement dependencies between tasks. However, there will be a hidden problem here, that is, the dependency relationship between tasks is established based on the code code defined by the workflow. When constructing a complex task flow based on data warehouse layering, such as ods->dwd->ads->xxxx, if the user mistakenly deletes an upstream ods task and recreates the task, then the The dependencies for will not be built correctly because the code has changed despite the same task name. To solve this problem, we refactored the task dependency mechanism. Instead, establish dependencies based on task names, and perform rule validation when creating tasks to ensure that task names are unique across the cluster. In this way, even if the user accidentally deletes and recreates the task, the dependencies can still be mounted correctly.

With this improvement, we ensure the accuracy and stability of the dependencies between tasks throughout the scheduling system. Users no longer need to worry about the problem that dependencies cannot be established due to accidental deletion. In this way, users can build complex task flows more confidently, ensuring that dependencies between tasks are correctly established and maintained.

Support configuration based schedule generation

In actual work, we will also encounter some users who do not understand SQL, but still need to process and use data. In order to satisfy this part of users, we implemented a configuration-based schedule generation mode. The design idea of this mode is to allow users to define the process of data processing and processing through simple configuration without writing complex SQL statements.

file

By selecting configurations, including information such as tables, table relationships, filter conditions, and targets, our platform can automatically generate corresponding SQL statements based on the information selected by users. Then, according to the generated SQL statement, the above-mentioned steps of SQL parsing and scheduling configuration are performed to complete the configuration process of the scheduling task.

This configuration-based scheduling generation mode enables users who do not understand the SQL language to easily complete data processing and use, which greatly reduces the learning and use threshold. At the same time, it also improves development efficiency and reduces the possibility of errors. This design allows us to better cover users of different technical levels and meet their needs.

Ambari Management DS support

Unified management of parameter configuration and comparison of multiple versions

As a distributed application, DolphinScheduler needs to be synchronized among the nodes every modification to the configuration file. Not only is this process cumbersome, but historical changes cannot be traced back. Especially for the configuration of online services, a slight change may lead to the paralysis of services. In view of these problems, our team urgently needs a tool that can provide clear configuration item display, configuration rollback, and historical version comparison.

Ambari provides us with such a solution. Therefore, we integrate DolphinScheduler with Ambari by writing a plug-in to achieve unified management of all configuration items. This integrated method not only reduces the work pressure of the operation and maintenance personnel, but also enhances the stability of the service, bringing a strong guarantee for the robustness and reliability of the system.

Unified management of configuration items: file

Comparison of modified versions of configuration items: file file

Visual one-click cluster start and stop

On the management interface of Ambari, users can clearly view the status of the entire cluster, including the running status of each node. Through a simple one-click operation, users can start or stop the entire cluster deployed on multiple server nodes, which makes cluster management more intuitive and convenient.

file

Support for automatic service failure "self-healing"

On the Ambari platform, it is very simple to implement "self-healing" of the DolphinScheduler service. Just select the DolphinScheduler service that needs to be monitored on the platform, and you can use Ambari's Service Auto Start function to easily implement it. When the DolphinScheduler service is down due to some abnormal conditions, Service Auto Start will automatically and continuously try to restart the service, thus ensuring the high availability of the DolphinScheduler service and greatly improving the stability of the system.

file

End-to-end scheduling component availability monitoring WatchDog

Although DolphinScheduler already has a task fault-tolerant mechanism and a high-availability mechanism, in the actual execution process, it still needs to involve many external services, such as Hiveserver2, ClickHouse, etc. In such a complex environment, the main goal of the operation and maintenance personnel is not to require the system to be 100% fault-free, but to get an immediate response when a problem occurs, so that the operation and maintenance personnel can intervene as soon as possible and solve the problem. Solve the problem. Therefore, Ziroom developed an end-to-end full-link usability testing task—WatchDog.

The goal of WatchDog is to monitor the availability of the entire scheduling process and immediately alert the operation and maintenance staff if a problem is found. This design greatly improves the response speed of the problem and the stability of the system, reduces the burden on the operation and maintenance personnel, and enhances the ability to control the operating status of the entire system. file

Internal core logic "buried point"

In the early stage of actual use of DolphinScheduler (version 2.0.6), it is no secret that many problems were encountered. However, these problems did not discourage the team, because the team also fully understands that as a relatively young scheduling product, it is very normal for DolphinScheduler to have various problems. Therefore, we decided to fix these problems to ensure that DolphinScheduler can "settle down" smoothly.

During this process, many challenges were encountered. For example, when the scheduling system is running, hundreds or thousands of tasks may be executed in parallel at the same time, and each workflow corresponds to a thread object and a built-in storage queue. In addition, there are many background threads and logic intertwined with remote Netty communication trigger flow, which brings great difficulties to the positioning of the problem.

To solve this problem, we developed a special exception annotation. After the method is marked with this annotation, when an exception occurs, it will record the input parameters, output parameters, thread name, execution time, server node where it is located, and detailed exception stack information. Then, through corporate WeChat and telephone alarms, the information is immediately notified to the operation and maintenance personnel, so as to realize timely positioning and processing of problems. Therefore, we annotate and cover the key links of the scheduling process, such as start, stop, complement, fault tolerance, and timing scheduling. This approach greatly increases our control over the operating state of the system, enabling us to identify and resolve problems more quickly.

JVM parameter optimization monitoring

The key to JVM tuning is to seek a "balance". We adjust the memory size to balance the frequency and duration of garbage collection.

Our optimization goals are: first, to ensure the throughput of the service; second, to reduce the pause time of the service. At the same time, we corrected the code that may have memory leaks in DolphinScheduler, especially changed some static attributes to local variables to avoid memory leaks. This is because local variables only work in code blocks, while static properties are related to class objects. In the JVM, the unloading conditions of classes are very strict. Our application is loaded by the system class loader, so the loaded classes will not be unloaded. This means that if an object is hung on the class loader, the object will not be released. In contrast, local variables are deallocated shortly after use. Therefore, our goal is to avoid static marking of any collection or heavyweight objects in the JVM process.

The following is the JVM configuration we made in terms of JVM parameter optimization for readers' reference:

XX:+UseG1GC: Use the G1 garbage collector.
XX:MaxGCPauseMillis=500: Set the maximum GC pause time to 500 milliseconds.
XX:+PrintGCDetails
XX:+PrintGCTimeStamps
XX:+PrintGCCause: Print GC details, timestamp and GC reason.
XX:+UseGCLogFileRotation
XX:NumberOfGCLogFiles=10
XX:GCLogFileSize=10M: Enable rotation of GC log files, keep 10 files, each file size is 10 megabytes.
XX:+HeapDumpOnOutOfMemoryError、
XX:+HeapDumpBeforeFullGC: Generate a heap dump file when an OutOfMemoryError occurs, and generate a heap dump file before performing a Full GC.
Xmx16g, -Xms16g: Set the maximum and initial size of the heap to 16GB.
XX:MetaspaceSize=512m
XX:MaxMetaspaceSize=512m: Set the initial and maximum size of the metaspace to 512MB.

In addition, we also use javaAgent to collect JVM-related indicators and report them to Prometheus to establish more comprehensive monitoring.

Authority control

Our management of scheduling authority is mainly realized by controlling the HQL task execution queue. First, during the login process, the scheduling platform will authenticate the user through LDAP. The server will obtain the department where the user's LDAP is located, and create a corresponding project on the scheduling platform according to the department name. In other words, the user's department in LDAP will be mapped to an item space on DolphinScheduler.

Second, each project is associated with a specific Hive data source. Each data source differs in their queue settings. Since the tasks of different departments will be submitted to the corresponding Yarn queues due to the associated data sources, this setting enables the resource usage of each department to be priced and effectively managed.

In this way, we ensure the strict control of scheduling authority, and at the same time realize the accurate calculation and efficient management of departmental resources.

user income

Meet the data processing and use needs of business personnel, analysts, and product BPs: through the configuration-based scheduling generation mode and the function of automatically generating SQL, we can meet the data requirements of business personnel, analysts, and product BPs who do not understand the SQL language. Processing and use requirements. They can define the data processing process through simple configuration without writing complex SQL statements. This enables them to quickly and easily process and use data, increasing their productivity.
Meet the needs of operation and maintenance personnel to maintain complex data scheduling: For operation and maintenance personnel, maintaining complex data scheduling tasks is a challenging task. However, through our scheduling platform, operation and maintenance personnel can manage and maintain these scheduling tasks more conveniently. The platform provides a visual operation interface and rich functions, such as configuration rollback and historical version comparison, etc., enabling operation and maintenance personnel to easily manage and maintain tasks. At the same time, the end-to-end availability monitoring tool WatchDog can also detect system failures in time, improving the stability and reliability of the scheduling system.

By meeting the data processing and usage needs of business personnel, analysts, and product BPs, as well as the needs of operation and maintenance personnel to maintain complex data scheduling, our upgraded scheduling platform based on Apache DolphinScheduler can provide comprehensive support for each role and promote business smooth and efficient operation and maintenance.

This article is supported by Beluga Open Source Technology !