1, ready to work

1.1 virtual machine to take a hostname. And the configuration hosts. If you want to win and do joint development of the case, and win the hosts file, do the same domain name mapping.

# 127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
# ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.241.128 master

1.2 do Free Density

[root@master opt]# ssh-keygen -t rsa

[root@master opt]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@master

[root@master opt]# ssh root@master

1.3 Installation dk

vim /etc/profile

export JAVA_HOME=/usr/local/java/jdk1.8.0_221
export CLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar
export PATH=$JAVA_HOME/bin:$PATH

2、hadoop

Profiles

hadoop-env.sh

export JAVA_HOME=/usr/local/java/jdk1.8.0_221

core-site.xml

<property>
  <name>fs.defaultFS</name>
  <value>hdfs://master:9000</value>
</property>

<property>
  <name>hadoop.tmp.dir</name>
  <value>/opt/hadoop-2.7.7/tmp</value>
</property>

hdfs-site.xml

  <property>
     <name>dfs.replication</name>
     <value>1</value>
  </property>

  <property>
     <name>dfs.permissions</name>
     <value>false</value>
  </property>

maper-site.xml

  <property>
     <name>mapreduce.framework.name</name>
     <value>yarn</value>
  </property>

yarn-site.xml

  <property>
     <name>yarn.resourcemanager.hostname</name>
     <value>master</value>
  </property>


  <property>
     <name>yarn.nodemanager.aux-services</name>
     <value>mapreduce_shuffle</value>
  </property>

Environment Variables

export HADOOP_HOME=/opt/hadoop-2.7.7
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

Format namenode. Once formatted, the formatting can not be repeated.

[root@master ~]# hdfs namenode -format

start up

start-all.sh

[root@master ~]# start-all.sh 
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Starting namenodes on [master]
master: starting namenode, logging to /opt/hadoop-2.7.7/logs/hadoop-root-namenode-master.out
localhost: starting datanode, logging to /opt/hadoop-2.7.7/logs/hadoop-root-datanode-master.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /opt/hadoop-2.7.7/logs/hadoop-root-secondarynamenode-master.out
starting yarn daemons
starting resourcemanager, logging to /opt/hadoop-2.7.7/logs/yarn-root-resourcemanager-master.out
localhost: starting nodemanager, logging to /opt/hadoop-2.7.7/logs/yarn-root-nodemanager-master.out
[root@master ~]# jps
4081 DataNode
4853 Jps
4728 NodeManager
4409 ResourceManager
3947 NameNode
4252 SecondaryNameNode

验证

http://master:50070/ ，这是 hdfs（hadoop）的 web页面。

http://master:8088/ ，是yarn 资源调度的web页面。

开启历史服务器

yarn.site.xml

  <property>
    <name>yarn.log-aggregation-enable</name>
    <value>true</value>
  </property>

maper-site.xml

  <!-- 设置jobhistoryserver 没有配置的话 history入口不可用 -->
  <property>
    <name>mapreduce.jobhistory.address</name>
    <value>master:10020</value>
  </property>

  <!-- 配置web端口 -->
  <property>
    <name>mapreduce.jobhistory.webapp.address</name>
    <value>master:19888</value>
  </property>

  <!-- 配置正在运行中的日志在hdfs上的存放路径 -->
  <property>
    <name>mapreduce.jobhistory.intermediate-done-dir</name>
    <value>/history/done_intermediate</value>
  </property>

  <!-- 配置运行过的日志存放在hdfs上的存放路径 -->
  <property>
    <name>mapreduce.jobhistory.done-dir</name>
    <value>/history/done</value>
  </property>

启动， mr-jobhistory-daemon.sh start historyserver ，网页端口号是 19888

参考 https://blog.csdn.net/xiaoduan_/article/details/79689882

Hadoop remember a pseudo-distributed build environment (Linux)

1, ready to work

2、hadoop

Guess you like