Pseudo (implemented on one host) distributed application

1. Configure the cluster
2. Working together to cluster
3. Execution case
   1. Configure hadoop-env.sh file

Write the above path into a file, either vi or NotePad++ file can be used.

Add in /etc/hadoop/hadoop-env.sh

export JAVA_HOME=/opt/module/jdk1.8.0_144

Placement core-site.xml

<property>
<name>fs.defaultFS</name>
    <value>hdfs://hadoop101:9000</value>
</property>

<!-- 指定Hadoop运行时产生文件的存储目录 -->
<property>
    <name>hadoop.tmp.dir</name>
    <value>/opt/module/hadoop-2.7.2/data/tmp</value>
</property>

Configure hdfs-site.xml

<property>
    <name>dfs.replication</name>
    <value>1</value>
</property>

2. Start the cluster

Namenode of the master node (cannot always be formatted, the cluster id will change)

bin/hdfs namenode -format

Start the NameNode

sbin/hadoop-demon.sh start namenode

Working together DataNode

sbin/hadoop-daemon.sh start datanode

Investigate the cluster

jps

Be careful: jps is a JDK command, Hadoop is written in java, and JDK is also required.

http://hadoop101:50070/dfshealth.html#tab-overview

Pseudo-distributed startup hdfs (Hadoop Dir file System)
create file input to


check whether the uploaded file is correct

bin/hdfs dfs -ls  /user/dev1/input/
bin/hdfs dfs -cat  /user/dev1/input/word.txt

Run MapReduce

bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar wordcount /user/dev1/input/ /user/dev1/output

查看成功的结果：
bin/hdfs dfs -cat /user/dev1/output/*

Placement yarn-site.xml

Operate under dev1

sudo vim yarn-site.xml

Add to the file:

<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>

<!-- 日志保留时间设置7天 -->
<property>
<name>yarn.log-aggregation.retain-seconds</name>
<value>604800</value>
</property>

Start NodeManager, ResourceManager and HistoryManager

sbin/yarn-daemon.sh start resourcemanager
sbin/yarn-daemon.sh start nodemanager
sbin/mr-jobhistory-daemon.sh start historyserver

jps
6150 NodeManager
5912 ResourceManager
6284 JobHistoryServer
6317 Jps

Execute WordCount

bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar wordcount /user/dev1/input /user/dev1/output

All the code is attached to the
history

    1  java -version
    2  ls
    3  cd etc
    4  ls
    5  cd hadoop/
    6  ls
    7  vi yarn-site.xml 
    8  sbin/yarn-daemon.sh start resourcemanager
    9  sbin/yarn-daemon.sh start nodemanager
   10  sbin/mr-jobhistory-daemon.sh start historyserver
   11  sbin/yarn-daemon.sh start resourcemanager
   12  cd ..
   13  ls
   14  sbin/yarn-daemon.sh start resourcemanager
   15  sbin/yarn-daemon.sh start nodemanager
   16  sbin/mr-jobhistory-daemon.sh start historyserver
   17  jps
   18  bin/hdfs dfs -rm -R /user/dev1/output
   19  bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-example

s-2.7.2.jar wordcount /user/dev1/input /user/dev1/output