Hadoop-3-additional configuration

Hadoop-3-additional configuration

1. Configure the MapReduce job history server

[1] Modify the configuration file

 etc/hadoop/mapred-site.xml

<!-- 配置MapReduce任务历史服务器主机地址:端口，端口号默认为10020 -->
<property>
    <name>mapreduce.jobhistory.address</name>
    <value>MapReduce job历史服务器主机地址:端口</value>
</property>
<!-- 配置MapReduce任务历史服务器Web页面主机地址:端口，端口号默认为19888 -->
<property>
    <name>mapreduce.jobhistory.webapp.address</name>
    <value>MapReduce job历史服务器Web页面主机地址:端口</value>
</property>

[2] Start MapReduce JobHistoryServer

 sbin/mr-jobhistory-daemon.sh start historyserver

[3] Verification

Click on an Application Tracking UIcolumn on the YARN ResourceManager management pageHistory

【4】Notes

Need to close all Hadoop resources (NameNode, DataNode, ResourceManager, NodeManager, etc.)

2. Turn on the historical log aggregation function

[1] Modify the configuration file

 etc/hadoop/yarn-site.xml

<!-- 启用日志聚合。默认关闭状态（false） -->
<property>
    <name>yarn.log-aggregation-enable</name>
    <value>true</value>
</property>
<!-- 设置日志保留时长，单位是秒。默认是-1，表示禁用 -->
<property>
    <name>yarn.log-aggregation.retain-seconds</name>
    <value>日志保留时长</value>
</property>

[2] Verification

Click YARN ResourceManager an Application Management page of the Tracking UIcolumns Historyin the Logscolumnlogs

【3】Notes

 ① You need to configure and start the JobHistoryServer of MapReduce first

 ② You need to close all Hadoop resources (NameNode, DataNode, ResourceManager, NodeManager, JobHistoryServer, etc.)