Apache Spark integration Yarn error or can not view the log solution

Integration Yarn error or can not view the log solution

The first step: Modify hadoop of yarn-site.xml

vim /export/servers/hadoop/etc/hadoop/yarn-site.xml

       <property>
                <name>yarn.resourcemanager.hostname</name>
                <value>node01</value>
        </property>
        <property>
                <name>yarn.nodemanager.aux-services</name>
                <value>mapreduce_shuffle</value>
        </property>
<!-- 关闭YARN内存检查 -->
	<property>
                <name>yarn.nodemanager.pmem-check-enabled</name>
                 <value>false</value>
        </property>
        <property>
                    <name>yarn.nodemanager.vmem-check-enabled</name>
                    <value>false</value>
        </property>
	<!-- 如果开启如下配置则需要开启Spark历史服务器
	<property>
                <name>yarn.log-aggregation-enable</name>
                <value>true</value>
        </property>
        <property>
                <name>yarn.log-aggregation.retain-seconds</name>
                <value>604800</value>
        </property>
        <property>
                <name>yarn.log.server.url</name>
                <value>http://node01:19888/jobhistory/logs</value>
        </property>
	-->

Step Two: distribute the configuration file to each node

The third step: restart the Hadoop services

/export/servers/hadoop/sbin/stop-dfs.sh
/export/servers/hadoop/sbin/stop-yarn.sh
/export/servers/hadoop/sbin/start-dfs.sh
/export/servers/hadoop/sbin/start-yarn.sh

YARN server consolidation history and the history of server Spark

1. Turn YARN history server configuration and start

/export/servers/hadoop/sbin/mr-jobhistory-daemon.sh start historyserver

2. Spark configure the log server [reference to the following chapters]

3. Modify the spark-defaults.conf

vim /export/servers/spark/conf/spark-defaults.conf

spark.yarn.historyServer.address node01:4000

4. Start Spark HistoryServer Service

/export/servers/spark/sbin/start-history-server.sh

5. If more dependent Jar package can be uploaded to the HDFS and told to fetch YARN

Disposed in the spark-default.conf

spark.yarn.jars = hdfs://node01:8020/sparkjars/*

After you configure each node will go on HDFS downloaded and cached
each node if you do not configure Spark program will start to distribute packaged Spark_HOME

Spark history log server configuration

By default, the Spark program has finished running, close the window, you can not see the log of the Web UI (4040), and by HistoryServer can provide a service by reading the log file so that we can run at the end of the program, still can View during operation

1. Modify the file name

cd /export/servers/spark/conf
cp spark-defaults.conf.template spark-defaults.conf

2. modify the configuration

vim spark-defaults.conf

spark.eventLog.enabled true
spark.eventLog.dir hdfs://node01:8020/sparklog

3. Note: The directory on the need to manually create hdfs

hadoop fs -mkdir -p /sparklog

4. Modify spark-env.sh

vim spark-env.sh
export SPARK_HISTORY_OPTS="-Dspark.history.ui.port=4000 -Dspark.history.retainedApplications=3 -Dspark.history.fs.logDirectory=hdfs://node01:8020/sparklog"

5. Synchronize Files

scp -r /export/servers/spark/conf/ @node02:/export/servers/spark/conf/
scp -r /export/servers/spark/conf/ @node03:/export/servers/spark/conf/

6. Restart the cluster

/export/servers/spark/sbin/stop-all.sh
/export/servers/spark/sbin/start-all.sh 

7. Start a log server on the master

/export/servers/spark/sbin/start-history-server.sh

8. View the history log port in 4000 (if the load does not come out to try to change the browser)

http://node01:4000/

9. If you have write access to Hadoop HDFS questions:

org.apache.hadoop.security.AccessControlException
Solution: Add the following configuration hdfs-site.xml, the closure authority verification

<property>
        <name>dfs.permissions</name>
        <value>false</value>
</property>