Spark on Yarn运行的时候Job History Server的配置
1.配置如下:yarn-site.xml
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.log.server.url</name>
<value>http://ZQ0:19888/jobhistory/logs</value>
</property>
2.重启yarn服务及mr job history服务
3. 配置spark的conf/spark-defaults.conf文件内容
vim conf/spark-defaults.conf
spark.yarn.historyServer.address
http://ZQ0:18080
4. 重新运行spark on yarn应用即可
./bin/spark-submit
–class com.ZQ.bigdata.spark.app.core.PVAndUVSparkCore
–master yarn
–deploy-mode cluster
–conf spark.yarn.submit.waitAppCompletion=false
/home/ZQ0/logs-analyzer-1.0.jar
备注:在提交任务之前如下进程需要启动好。
Namenode
DateNode
HistoryServer
job HistoryServer
ResourceManager
NodeManager
寄语专区:
转载请注明出处。