This article original, For reprint, please indicate the original author and link
Now the configuration of the node
1, yarn single-node configuration management
We need to configure mapread-site.xml
Configure parameters as follows:
etc/hadoop/mapred-site.xml:
#mapread-site.xml <configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> </configuration> etc/hadoop/yarn-site.xml: #和yarn-site.xml <configuration> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> </configuration>
2, the configuration of the yarn management and HA
We need to configure a yarn-site.xml file
Here is the sample of minimal setup for RM failover.
<property> <name>yarn.resourcemanager.ha.enabled</name> <value>true</value> </property> <property> <name>yarn.resourcemanager.cluster-id</name> <value>cluster1</value> </property> <property> <name>yarn.resourcemanager.ha.rm-ids</name> <value>rm1,rm2</value> </property> <property> <name>yarn.resourcemanager.hostname.rm1</name> <value>node03</value> </property> <property> <name>yarn.resourcemanager.hostname.rm2</name> <value>node04</value> </property> <property> <name>yarn.resourcemanager.zk-address</name> <value>node02:2181,node03:2181,node04:2181</value> </property>
3, those few specific process node will be built in its own node resourcemanager as where you need these two nodes to each other to distribute keys, free to achieve mutual secret key login
node03
[root@node03 ~]# ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
Realization of their own free keys Login
[root@node03 ~]# cd .ssh/ [root@node03 .ssh]# ll 总用量 20 -rw-r--r-- 1 root root 616 9月 23 11:35 authorized_keys -rw------- 1 root root 672 10月 4 15:57 id_dsa -rw-r--r-- 1 root root 601 10月 4 15:57 id_dsa.pub -rw-r--r-- 1 root root 171 9月 23 11:26 known_hosts -rw-r--r-- 1 root root 616 9月 23 11:34 node01.pub [root@node03 .ssh]# cat id_dsa.pub >> authorized_keys
The secret key issue in node03 node04 node and renamed
[root@node03 .ssh]# scp id_dsa.pub node04:`pwd`/node03.pub
After performing additional command node in
[root@node04 .ssh]# cat node03.pub >> authorized_keys
The same node04 and node03 the same command and his secret key documents issued Node03 and 03 keys on node04 file additional documents to verify their
4, start the cluster step
Summary:
Start: node01: 1 ZooKeeper 2 HDFS (Note that there is a script not to use, Start-All) Start- dfs.sh If nn and nn2 does not start, you need node06, node07 were manually start: hadoop - daemon.sh Start the NameNode 3 Start- yarn.sh (start nodemanager) 4 at nodes 03 and 04 respectively execute the script: yarn- daemon.sh start the ResourceManager the UI visit: ip: 8088 stop: node01: sTOP - dfs.sh node01: sTOP - yarn.sh (stop nodemanager) node03, node04: the Yarn -daemon.sh the ResourceManager sTOP (stop resourcemanager)
First start zk cluster; command: zkServer.sh start
Because mr hdfs is the basis of which, it is on the primary node first start hdfs command: start-dfs.sh
hdfs cluster startup:
Start Node Manager on the primary node, this process will be started on each of the node DataNode
[root@node01 hadoop]# start-yarn.sh
Noted herein does not start, the node needs to configure two separate resourcemanager go to start it
[root@node03 .ssh]# yarn-daemon.sh start resourcemanager starting resourcemanager, logging to /opt/sxt/hadoop-2.6.5/logs/yarn-root-resourcemanager-node03.out
[root@node04 .ssh]# yarn-daemon.sh start resourcemanager starting resourcemanager, logging to /opt/sxt/hadoop-2.6.5/logs/yarn-root-resourcemanager-node04.out
After by ss -nal command to view port 8088 is the default communication ports yarn and browser,
The 3888 election is the port zk cluster mechanism used, zk 2888 is the port used for internal cluster communication, 2181 is a single-port server and client communications
Browser Access yarn
To use its computing power:
First create yourself and upload a file
[root@node01 hadoop-2.6.5]# for i in `seq 100000`;do echo "hello sxt $i" >> test.txt;done
[root@node01 software]# hdfs dfs -D dfs.blocksize=1048576 -put test.txt
Specific use of computing functions process
To find their own path hadoop installation
cd /opt/sxt/hadoop-2.6.5/share/hadoop/mapreduce
Test.txt word count to be
[root@node01 mapreduce]# hadoop jar hadoop-mapreduce-examples-2.6.5.jar wordcount /test.txt /wordcount
You can see the progress of the calculation process scroll bar
Background execution linux command line
First job execution execute map reduce jobs
Browser-side observation calculation has been completed
The command line to see the finished phenomenon
After waiting for the heart is completed, it can command in the command line
View hdfs files
hdfs dfs -cat / file viewing survey results
hdfs dfs -cat /wordcount/part-r-00000
result
Mr complete calculation of the basic functions! !