Conclusion: When the deployment mode is local, the master parameter is local; otherwise, the master parameter is Yarm.
Today I successfully configured the spark task in dolphinscheduler, but in the log of the instance task, I found that the parameters spark-submitin the spark task submission command --masterare always yarm, and there is no configuration that should be explained on the task configuration page, so I went to analyze the source code. .
1. First, locate the source code file path: dolphinscheduler-task-plugin\dolphinscheduler-task-spark\src\main\java.
2. You can take a look at SparkConstants.java first. There are some configuration constants defined in it. It will look familiar and facilitate later analysis.
3. Locate the populateSparkOptions function of SparkTask.java. Some key information has been annotated in the picture below
. There are two situations:
when deployMode!="local", the value in the args list --master yarn --deploy-mode deployMode
is deployMode=="local", when the args list The value in is. --master local
I won’t go into details about other configurations later. This article focuses on the --master parameter.
populateSparkOptionsAt the end of the function return args；
4. Locate the place where the populateSparkOptions function is called and see how the args parameter is integrated spark-submitinto the task submission command.
You can see that it uses spaces to connect the elements in the args list object to prepare for splicing into a submit command.