A new generation of big data task scheduling-Apache DolphinScheduler (incubator) has released version 1.3.3 with the contributions and efforts of more than 20 small partners in the community. 1.3.3 is a bug fix version of 1.3.x, which has been fixed this time. 1.3.2 More than 30 bugs were discovered, most of which were discovered by test partners in extreme environments. 1.3.x important fixes need to pay attention to the following 3:
3 important fixes:
1. Phenomenon: An exception was found on the worker node where the task was executed: Too many open files #3618
the reason:
[worker] Too many files will be opened because the task execution has been completed but the file handle has not been released.
[ api ] When deleting process instance cascade delete task log, log client does not close
solve:
https://github.com/apache/incubator-dolphinscheduler/pull/3075 https://github.com/apache/incubator-dolphinscheduler/pull/3971
2. Phenomenon: When the worker sends the execution result to the master , the master just hangs up. After restarting the master , the workflow and task status are always in progress, and the workflow cannot be stopped
Reason: The master and the worker interact asynchronously, and the task status is lost in extreme situations in the communication process.
solve:
Some links are changed to synchronization, please refer to the specific fix: https://github.com/apache/incubator-dolphinscheduler/pull/3748
3. Phenomenon : Under certain network conditions, when master submit task , netty communication cannot be carried out , task information cannot be sent to worker , after waiting a long time, a time out exception occurs , and then it will appear again after a while phenomenon. #3789
Solution: Add heartbeat detection between master and worker PR #3913
For other BUG FIX, please refer to the following:
[ #3615 ] [master] Successfully execute the task, but after the next task has not been submitted, stop the master and the workflow will fail
[ #4010 ] [master] When the condition node fails, the process status is failed after fault tolerance
[ # 3929 ] [master/worker] After the master and worker are fault-tolerant, the process instance fails after the task instance is successfully executed
[ #3994 ] [Sql] Cannot import the SQL file of sql/dolphinscheduler_mysql.sql
[ #3964 ] [api/ui] of sub_process Timeout warning is not effective
[ #3974 ] [api/worker] The child process does not inherit the workgroup of the parent process
[ #3966 ] [master] The timeout warning is not effective in the sub_process
[ #3958 ] [api] file and should not be in the following child The directory where the file authorization file is successfully created in the directory
[ #3618 ] [Staff] Since the task execution has been completed but the file handle has not been released, too many files will be opened.
[ #3615 ] [master] After the task is successfully executed but the next task has not been submitted, stop the master and the workflow will fail.
[ #3843 ] [api] When the update workflow definition name already exists, the prompt is not friendly.
[ #3924 ] [ui] When saving the workflow definition, if the backend verification is unsuccessful, the save pop-up window will not close.
[ #3789 ] [remote] Support netty heartbeat to solve the channel timeout problem.
[ #3836 ] [api] verifyProcessDefinitionName error message
[ #3702 ] [api] Re-uploading the resource file will not overwrite the original file. This button is invalid.
[ #3621 ] [master] After deleting the process instances that are being executed in batch, the master cannot get feedback from the staff.
[ #3617 ] [master] After the subtask is fault-tolerant, 2 task instances are generated. The state of the process instance Always execute
[ #3616 ] [master] After the host receives the result from the staff, the host stops. After the main server restarts, the workflow status is always executing
[ #3573 ] [dao] Potential level of unauthorized access
[ #3887 ] [ui] The English version of the resource file list and UDF list data is not displayed.
[ #3702 ] [api] Re-uploading the resource file will not overwrite the original file. This button is invalid.
[ #3487 ] [API] Creating a folder with multiple threads will result in multiple identical folders
[ #3493 ] [api]Workgroup management ui cannot display creation time and update time
[ #3549 ] [Server] [sqlTask ]The alias column in the query SQL does not take effect
[ #3719 ] [api] Task resources can not be deleted
[ #3548 ] [UI] The monitor meter is blank with the same pid error
[ #3707 ] [ui] If you select , You cannot cancel the batch delete function in the workflow definition and workflow instance pages
[ #3553 ] [ui] When editing the workflow, the selected connection line has no style change, but other connection lines become larger
[ #3238 ] [ docker] Unable to create folder in docker in standalone mode
[ #3713 ] [common] The catfile method stream is not closed
[ #3258 ] [api] Unable to get the creation time and update time, report DateTimeParseException
In addition to bug fixes, several functional improvements have been made this time. Refer to the following:
[ #3720 ] [ui] js email verification
[ #3843 ] [api] When the update workflow definition name already exists, the prompt is not friendly
[ #3131 ] [api] When a new tenant already exists, the prompt is incorrect
The 1.3.3 version has undergone a lot of testing by test partners, and has fixed many problems that may occur in extreme situations. It is highly recommended that you download and deploy version 1.3.3
In addition, please refer to 1.3.3 for specific changes: https://github.com/apache/incubator-dolphinscheduler/releases/tag/1.3.3
1.3.3 Download link : https://dolphinscheduler.apache.org/zh-cn/docs/release/download.html
Apache DolphinScheduler community introduction
Apache DolphinScheduler is a very diverse community. So far, there are more than 100 contributors from more than 30 different companies. There are nearly 4,000 WeChat group users.
Some user cases (part of the ranking first)
Already, more than 400 companies and scientific research institutions are using DolphinScheduler to handle various scheduling and timing tasks, and more than 1,200 companies have opened a trial of Dolphin Scheduler:
Apache DolphinScheduler 能力
- Associate tasks according to their dependencies in the form of DAG graphs, and visually monitor the running status of tasks in real time
- Support rich task types: Shell, MR, Spark, Flink, SQL (mysql, postgresql, hive, sparksql), Python, Http, Sub_Process, Procedure, etc.
- Supports workflow timing scheduling, dependent scheduling, manual scheduling, manual pause/stop/resume, and supports failure retry/alarm, recovery failure from designated nodes, Kill tasks, etc.
- Support workflow priority, task priority and task failover and task overtime alarm/failure
- Support workflow global parameters and node custom parameter settings
- Support online upload/download of resource files, management, etc., support online file creation and editing
- Support task log online viewing and scrolling, online download log, etc.
- Realize cluster HA and realize decentralization of Master cluster and Worker cluster through Zookeeper
- Support online viewing of Master/Worker cpu load, memory, cpu
- Support workflow running history tree/Gantt chart display, support task status statistics, process status statistics
- Supports complement
- Support multi-tenancy
- Support internationalization
Participate in open source-join Apache DolphinScheduler
The ways in which the DolphinScheduler community participates in contributions include:
- Documentation
- translation
- Sermon
- Q & A
- test
- Code
- Practice articles
- Meeting sharing etc.
Welcome to join the contributing team, join the open source community from submitting the first PR
-Find the "easy to fix" mark or some very simple issues (such as spelling mistakes, etc.), and get familiar with the submission process through the first PR. If you have any questions, please feel free to contact
Mail subscription method:
https://dolphinscheduler.apache.org/zh-cn/docs/development/subscribe.html
gitee:
https://gitee.com/dolphinscheduler/DolphinScheduler
github：
https://github.com/apache/incubator-dolphinscheduler
Official website
https://dolphinscheduler.apache.org
Finally, you are very welcome to try online http://106.75.43.194:8888/ !