目录
Hadoop部署
一、配置core-site.xml。代理主机
[doudou@hadoop101 module]$ cd hadoop-3.1.3/
[doudou@hadoop101 hadoop-3.1.3]$ cd etc/hadoop/
[doudou@hadoop101 hadoop]$ cd core-site.xml
[doudou@hadoop101 hadoop]$ vim /core-site.xml
<configuration>
<!-- 指定NameNode的地址 -->
<property>
<name>fs.defaultFS</name>
<value>hdfs://hadoop102:8020</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/opt/module/hadoop-3.1.3/data</value>
</property>
<property>
<name>hadoop.http.staticuser.user</name>
<value>doudou</value>
</property>
<property>
<name>hadoop.proxyuser.doudou.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.doudou.grups</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.doudou.users</name>
<value>*</value>
</property>
</configuration>
二、HDFS配置文件
配置hdfs-site.xml
[doudou@hadoop101 hadoop]$ vim hdfs-site.xml
<configuration>
<property>
<name>dfs.namenode.http-address</name>
<value>hadoop101:9870</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>hadoop103:9868</value>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
~
分发
[doudou@hadoop101 hadoop]$ xsync hdfs-site.xml
三、YARN配置
配置yarn-site.xml
[doudou@hadoop101 hadoop]$ vim yarn-site.xml
<configuration>
<!-- Site specific YARN configuration properties -->
<!-- 指定MR走shuffle -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<!-- 指定ResourceManager的地址-->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>hadoop103</value>
</property>
<!-- 环境变量的继承 -->
<property>
<name>yarn.nodemanager.env-whitelist</name>
<value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME</value>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>512</value>
</property>
<property>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>4096</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>4096</value>
</property>
<property>
<name>yarn.nodemanager.vmem-check-enabled</name>
<value>false</value>
</property>
</configuration>
[doudou@hadoop101 hadoop]$ xsync yarn-site.xml
四、MapReduce配置文件
配置mapred-site.xml
[doudou@hadoop101 hadoop]$ vim mapred-site.xml
<configuration>
<!-- 指定MapReduce程序运行在Yarn上 -->
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
[doudou@hadoop101 hadoop]$ xsync mapred-site.xml
五、配置workers
[doudou@hadoop101 ~]$ vim /opt/module/hadoop-3.1.3/etc/hadoop/workers
hadoop101
hadoop102
hadoop103
[doudou@hadoop101 hadoop]$ xsync workers
配置历史服务器
一、配置mapred-site.xml
[doudou@hadoop101 hadoop]$ vim mapred-site.xml
<!-- 历史服务器web端地址 -->
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>hadoop101:19888</value>
</property>
>mapreduce.framework.name</name>
<value>yarn</value>
</property>
[doudou@hadoop101 hadoop]$ xsync mapred-site.xml
配置日志的聚集
一、配置yarn-site.xml
[doudou@hadoop101 hadoop]$ vim yarn-site.xml
<!-- 开启日志聚集功能 -->
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<!-- 设置日志聚集服务器地址 -->
<property>
<name>yarn.log.server.url</name>
<value>http://hadoop101:19888/jobhistory/logs</value>
</property>
<!-- 设置日志保留时间为7天 -->
<property>
<name>yarn.log-aggregation.retain-seconds</name>
<value>604800</value>
</property>
[doudou@hadoop101 hadoop]$ xsync yarn-site.xml