CentOS 7 + Hadoop-2.7.2 hadoop environmental structures of (b) pseudo-distributed mode of operation

In hadoop environment to build (a) to quickly create the first virtual machine continues to build in the environment

Start HDFS

Cluster Configuration

1. Configuration: hadoop-env.sh

   [zyy@hadoop-00 hadoop-2.7.2]$ cd /opt/module/hadoop-2.7.2/etc/hadoop/
   [zyy@hadoop-00 hadoop]$ vim hadoop-env.sh
   
   

   25 modified line: export JAVA_HOME = / opt / module / jdk1.8.0_162
   

2. Arrangement: core-site.xml
   [zyy@hadoop-00 hadoop]$ vim core-site.xml

First opened

in

 <configuration>
</configuration>

Add about content between, note the following configuration files appear Hadoop = 00 should be replaced by your own hostname

<!-- 指定HDFS中NameNode的地址 -->
<property>
<name>fs.defaultFS</name>
    <value>hdfs://Hadoop-00:9000</value>  
</property>

<!-- 指定Hadoop运行时产生文件的存储目录 -->
<property>
	<name>hadoop.tmp.dir</name>
	<value>/opt/module/hadoop-2.7.2/data/tmp</value>
</property>

Modify the results:

3. Configuration: hdfs-site.xml

<!-- 指定HDFS副本的数量 -->
<property>
	<name>dfs.replication</name>
	<value>1</value>
</property>

启动 NameNode, DataNode

1. (The first time you start the formatting, do not always after formatting) format NameNode

[zyy@hadoop-00 hadoop]$ cd /opt/module/hadoop-2.7.2/
[zyy@hadoop-00 hadoop-2.7.2]$ bin/hdfs namenode -format

The results appear this success

2. 启动 NameNode, DataNode

[zyy@hadoop-00 hadoop-2.7.2]$ sbin/hadoop-daemon.sh start namenode
[zyy@hadoop-00 hadoop-2.7.2]$ sbin/hadoop-daemon.sh start datanode

3. View results start

[zyy@hadoop-00 hadoop-2.7.2]$ jps

4. web end view HDFS file system
   http://192.168.218.132:50070/dfshealth.html#tab-overview

Start YARN

Cluster Configuration

1. Configuration yarn-env.sh

[zyy@hadoop-00 hadoop-2.7.2]$ cd /opt/module/hadoop-2.7.2/etc/hadoop/
[zyy@hadoop-00 hadoop]$ vim yarn-env.sh 

Configure the look JAVA_HOME

export JAVA_HOME=/opt/module/jdk1.8.0_162

2. Located yarn-site.xml

Note hadoop-00 should be replaced by your own hostname

<!-- Reducer获取数据的方式 -->
<property>
 		<name>yarn.nodemanager.aux-services</name>
 		<value>mapreduce_shuffle</value>
</property>

<!-- 指定YARN的ResourceManager的地址 -->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>hadoop-00</value>
</property>

3. Configuration: mapred-env.sh

Placed JAVA_HOME

export JAVA_HOME=/opt/module/jdk1.8.0_162

4. :( configuration of mapred-site.xml.template renamed) mapred-site.xml and modify

[zyy@hadoop-00 hadoop]$ mv mapred-site.xml.template mapred-site.xml
[zyy@hadoop-00 hadoop]$ vim mapred-site.xml

<!-- 指定MR运行在YARN上 -->
<property>
		<name>mapreduce.framework.name</name>
		<value>yarn</value>
</property>

Start ResourceManager, NodeManager

1. Before you start make sure NameNode and has started DataNode
2. Start ResourceManager, NodeManager

[zyy@hadoop-00 hadoop]$ cd /opt/module/hadoop-2.7.2/
[zyy@hadoop-00 hadoop-2.7.2]$ sbin/yarn-daemon.sh start resourcemanager
[zyy@hadoop-00 hadoop-2.7.2]$ sbin/yarn-daemon.sh start nodemanager

3. View the results start
   
   YARN web page:
   http://192.168.218.132:8088/cluster
   

Server configuration history

To view the history of the operation of the program, you need to configure the server history.

1. Placed mapred-site.xml

[zyy@hadoop-00 hadoop]$ cd /opt/module/hadoop-2.7.2/etc/hadoop/
[zyy@hadoop-00 hadoop]$ vi mapred-site.xml

Increasing the file which follows:

<!-- 历史服务器端地址 -->
<property>
<name>mapreduce.jobhistory.address</name>
<value>Hadoop-00:10020</value>
</property>
<!-- 历史服务器web端地址 -->
<property>
    <name>mapreduce.jobhistory.webapp.address</name>
    <value> Hadoop-00:19888</value>
</property>

2. Start the server history

[zyy@hadoop-00 hadoop]$ cd /opt/module/hadoop-2.7.2/
[zyy@hadoop-00 hadoop-2.7.2]$ sbin/mr-jobhistory-daemon.sh start historyserver

3. View results starting
   
   web end view: http: //192.168.218.132: 19888 / jobhistory

Gather configuration log

Log aggregation concept: After the completion of the application is running, the program will run on HDFS log information is uploaded to the system.
Log aggregation benefits: you can easily view the details of the program to run, easy development and debugging.
Enable the log aggregation function, you need to restart NodeManager, ResourceManager and HistoryManager.

1. Close NodeManager, ResourceManager and HistoryManager

[zyy@hadoop-00 hadoop-2.7.2]$ sbin/mr-jobhistory-daemon.sh stop historyserver
[zyy@hadoop-00 hadoop-2.7.2]$ sbin/yarn-daemon.sh stop nodemanager
[zyy@hadoop-00 hadoop-2.7.2]$ sbin/yarn-daemon.sh stop resourcemanager

View is closed:

2. Located yarn-site.xml

[zyy@hadoop-00 hadoop-2.7.2]$ cd /opt/module/hadoop-2.7.2/etc/hadoop/
[zyy@hadoop-00 hadoop]$ vim yarn-site.xml

Increasing the file which follows:

<!-- 日志聚集功能使能 -->
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>

<!-- 日志保留时间设置7天 -->
<property>
<name>yarn.log-aggregation.retain-seconds</name>
<value>604800</value>
</property>

3. Start NodeManager, ResourceManager and HistoryManager

[zyy@hadoop-00 hadoop]$ cd /opt/module/hadoop-2.7.2/
[zyy@hadoop-00 hadoop-2.7.2]$ sbin/yarn-daemon.sh start resourcemanager
[zyy@hadoop-00 hadoop-2.7.2]$ sbin/yarn-daemon.sh start nodemanager
[zyy@hadoop-00 hadoop-2.7.2]$ sbin/mr-jobhistory-daemon.sh start historyserver

Here pseudo-distribution is complete!