Ambari integrates high version spark2.x framework without upgrading

1. Background introduction

Everyone knows that spark has implemented a series of more convenient and quick improvements after 2.x. Spark currently has been updated to the 2.3.1 version in the community. The author found that spark has significantly enhanced the functions of spark-ml after 2.2.x. The big data experiment environment used by the author is based on Ambari, the version is 2.2.2, and the corresponding HDP version is 2.4.x. The author found that this version does not fully support spark2.x, and naturally it is impossible to use all aspects of the function enhancements brought by spark2.x. So the question is, how to integrate the high version of spark framework into ambari?

 

2. Integration plan

1. Option 1: Two-way upgrade of ambari and HDP

One way is naturally to upgrade the version of ambari itself to 2.4 (the latest version has reached 2.6.x). This kind of major upgrade requires careful operation. It is recommended to refer to the official documents on the hortonworks document, otherwise it may cause the cluster to crash. At the same time, because the dependence requires scientific Internet access, it is recommended to establish a local repos source for upgrade operations. Some users whose version is too low may need to double jump operation (first upgrade to 2.2, then upgrade to higher version). The overall upgrade plan is mentioned briefly, not as a key introduction, the following introduces a more secure integration of ambari and spark2.x framework without upgrading.

2. Option 2: ambari and spark2.x are not upgraded and integrated

1. After the spark version is recompiled according to the hadoop version of the ambari cluster, it is decompressed to a certain node of the cluster (one or more, not too much, according to the node constraints of the actual submitted task). The version compiled by the author is spark-2.2.0-bin-hadoop2.7.

2. Modify the spark-env.sh file and add the following content: Note that the HADOOP_CONF_DIR of the HDP version is different from the apache version and the CDH version. It is the /hadoop/conf folder under the corresponding version number, not /etc/hadoop

*JAVA_HOME
export JAVA_HOME=/usr/local/app/jdk1.8.0_144
*HADOOP_CONF_DIR
export HADOOP_CONF_DIR=/usr/hdp/2.4.2.0-258/hadoop/conf

3. Copy core-site.xml and hdfs-site.xml to spark's conf directory. If you need to integrate spark with existing hive or use spark-sql with hive as an external data source, you also need to add hive-site.xml Copy the file into the conf directory.

4. Solve the problem of jersey package version conflict: the jersey dependency package version of the cluster hadoop-yarn is lower than the corresponding package version of spark, so the relevant dependency package of spark must be replaced with the corresponding version of hadoop. Specifically, copy the two dependency packages jersey-core-1.9.jar and jersey-client-1.9.jar in the /usr/hdp/2.4.2.0-258/hadoop-yarn/lib directory to the lib directory of spark. At the same time, remove the corresponding high-version package in spark's lib directory, and the modified spark's lib directory is as shown below:

5. After the above four steps, spark can already submit running tasks in local, standalone, and yarn client mode on ambari, but when executing tasks in yarn cluster mode, the following error will appear:

18/07/21 20:55:40 INFO yarn.Client: Application report for application_1531149311507_0180 (state: ACCEPTED)
18/07/21 20:55:41 INFO yarn.Client: Application report for application_1531149311507_0180 (state: ACCEPTED)
18/07/21 20:55:42 INFO yarn.Client: Application report for application_1531149311507_0180 (state: ACCEPTED)
18/07/21 20:55:43 INFO yarn.Client: Application report for application_1531149311507_0180 (state: FAILED)
18/07/21 20:55:43 INFO yarn.Client: 
	 client token: N/A
	 diagnostics: Application application_1531149311507_0180 failed 2 times due to AM Container for appattempt_1531149311507_0180_000002 exited with  exitCode: 1
For more detailed output, check application tracking page:http://bdp03.szmg.com.cn:8088/cluster/app/application_1531149311507_0180Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_e22_1531149311507_0180_02_000001
Exit code: 1
Exception message: /hadoop/yarn/local/usercache/root/appcache/application_1531149311507_0180/container_e22_1531149311507_0180_02_000001/launch_container.sh: line 22: $PWD:$PWD/__spark_conf__:$PWD/__spark_libs__/*:$HADOOP_CONF_DIR:/usr/hdp/current/hadoop-client/*:/usr/hdp/current/hadoop-client/lib/*:/usr/hdp/current/hadoop-hdfs-client/*:/usr/hdp/current/hadoop-hdfs-client/lib/*:/usr/hdp/current/hadoop-yarn-client/*:/usr/hdp/current/hadoop-yarn-client/lib/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure: bad substitution

Stack trace: ExitCodeException exitCode=1: /hadoop/yarn/local/usercache/root/appcache/application_1531149311507_0180/container_e22_1531149311507_0180_02_000001/launch_container.sh: line 22: $PWD:$PWD/__spark_conf__:$PWD/__spark_libs__/*:$HADOOP_CONF_DIR:/usr/hdp/current/hadoop-client/*:/usr/hdp/current/hadoop-client/lib/*:/usr/hdp/current/hadoop-hdfs-client/*:/usr/hdp/current/hadoop-hdfs-client/lib/*:/usr/hdp/current/hadoop-yarn-client/*:/usr/hdp/current/hadoop-yarn-client/lib/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/${hdp.version}/hadoop/lib/hadoop-lzo-0.6.0.${hdp.version}.jar:/etc/hadoop/conf/secure: bad substitution

	at org.apache.hadoop.util.Shell.runCommand(Shell.java:576)
	at org.apache.hadoop.util.Shell.run(Shell.java:487)
	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:753)
	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:303)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Check the yarn log, locate the problem in the following statement: Cannot find the class

Error: Could not find or load main class org.apache.spark.deploy.yarn.ApplicationMaster

Solution: 
1) Add support for HDP version in spark-default.xml under spark conf file 
spark.driver.extraJavaOptions -Dhdp.version=2.4.2.0-258
spark.yarn.am.extraJavaOptions -Dhdp.version=2.4 .2.0-258

Note that the version number 2.4.2.0-258 here is the corresponding folder name under /usr/hdp, not the corresponding hadoop version number for executing hadoop version!

2) Replace the hdp.version information with the current version in the mapred-site.xml folder under HADOOP_CONF_DIR, and replace it in batches with sid:

sed -i 's/${hdp.version}/2.4.2.0-258/g'  mapred-site.xml

Three. Summary

Write a wordcount program to test spark on yarn cluster mode:

/usr/local/app/spark-2.2.0-bin-hadoop2.7/bin/spark-submit \
--class com.szmg.SparkWordCount \
--master yarn \
--deploy-mode cluster \
--driver-memory 2G \
--num-executors 4 \
--executor-memory 4G \
--executor-cores 5 \
--conf spark.app.coalesce=1 \
/usr/local/app/spark_test_projects/word_count/jar/scalaProject.jar  \
/testdata/README.md \
/testdata/output2

Submit to yarn for execution: success!

So far, the compatibility of ambari with any version of high version of spark is realized. I tested spark2.2.x and spark2.3.x, and successfully realized the spark on yarn cluster mode, thus realizing the ambari cluster for different versions of spark stand by.