New data node
0. demand
with the growth of business, the increasing amount of data, the capacity of the original data nodes can no longer meet the needs of data storage, data need to dynamically add a new node in the original cluster basis.
1. Environment Preparation
(1) on hadoop03 host then clone a hadoop04 host
(2) modify the IP address and host name
(3) delete the original files retain HDFS file system (/hadoop/module/hadoop-2.7.7/data and log)
(4) Source profiles about
[linyouyi@hadoop04 hadoop-2.7.7]$ source /etc/profile
2. The new service node specific steps
(1) to start directly DataNode, can be linked to the cluster
[linyouyi@hadoop04 hadoop-2.7.7]$ sbin/hadoop-daemon.sh start datanode [linyouyi@hadoop04 hadoop-2.7.7]$ sbin/yarn-daemon.sh start nodemanager
(2) upload files on hadoop04
[linyouyi@hadoop04 hadoop-2.7.7]$ hadoop fs -put /hadoop/module/hadoop-2.7.7/LICENSE.txt /
(3) If the data is not balanced, you can use the command rebalance cluster
[linyouyi@hadoop04 hadoop-2.7.7]$ ./start-balancer.sh starting balancer, logging to /hadoop/module/hadoop-2.7.7/logs/hadoop-linyouyi-balancer-hadoop01.out Time Stamp Iteration# Bytes Already Moved Bytes Left To Move Bytes Being Moved
(4) browser to view datanodes
Retired data node
Add whitelist
added to the white list of host nodes are allowed to access NameNode, the host node is not in the whitelist, it will be exited.
Specific steps whitelist is configured as follows:
(1) create a file at the NameNode dfs.hosts directory /hadoop/module/hadoop-2.7.7/etc/hadoop
[linyouyi @ hadoop01 hadoop] $ pwd /hadoop/module/hadoop-2.7.7/etc/ hadoop [linyouyi @ hadoop01 hadoop] $ Touch dfs.hosts [linyouyi @ hadoop01 hadoop] $ vi dfs.hosts # add the following host name ( do not add hadoop04) hadoop01 hadoop02 hadoop03
(2) increase in the NameNode dfs.hosts property profile hdfs-site.xml
<property> <name>dfs.hosts</name> <value>/hadoop/module/hadoop-2.7.7/etc/hadoop/dfs.hosts</value> </property>
(3) document distributed to each node
[linyouyi@hadoop01 hadoop]$ scp hdfs-site.xml linyouyi@hadoop02:/hadoop/module/hadoop-2.7.7/etc/hadoop/ [linyouyi@hadoop01 hadoop]$ scp hdfs-site.xml linyouyi@hadoop03:/hadoop/module/hadoop-2.7.7/etc/hadoop/ [linyouyi@hadoop01 hadoop]$ scp hdfs-site.xml linyouyi@hadoop04:/hadoop/module/hadoop-2.7.7/etc/hadoop/
(4) Refresh NameNode
[linyouyi@hadoop01 hadoop-2.7.7]$ hdfs dfsadmin -refreshNodes
Refresh nodes successful
(5) update the node ResourceManager
[linyouyi@hadoop01 hadoop-2.7.7]$ yarn rmadmin -refreshNodes 17/06/24 14:17:11 INFO client.RMProxy: Connecting to ResourceManager at hadoop02/192.168.1.103:8033
(6) View datanodes on the web browser
(7) If the data is not balanced, you can use the command rebalance cluster
[linyouyi@hadoop01 sbin]$ ./start-balancer.sh starting balancer, logging to /hadoop/module/hadoop-2.7.7/logs/hadoop-linyouyi-balancer-hadoop01.out Time Stamp Iteration# Bytes Already Moved Bytes Left To Move Bytes Being Moved
Blacklist retired
in the top of the host blacklist will be forced to quit.
1. Create dfs.hosts.exclude file in the directory NameNode of /hadoop/module/hadoop-2.7.7/etc/hadoop
[linyouyi@hadoop01 hadoop]$ pwd /hadoop/module/hadoop-2.7.7/etc/hadoop [linyouyi@hadoop01 hadoop]$ touch dfs.hosts.exclude [linyouyi@hadoop01 hadoop]$ vi dfs.hosts.exclude
Add the following host name (node to retire)
hadoop04
2. Increase in the NameNode dfs.hosts.exclude property profile hdfs-site.xml
<property> <name>dfs.hosts.exclude</name> <value>/hadoop/module/hadoop-2.7.7/etc/hadoop/dfs.hosts.exclude</value> </property>
3. Refresh NameNode, refresh ResourceManager
[linyouyi@hadoop01 hadoop-2.7.7]$ hdfs dfsadmin -refreshNodes Refresh nodes successful [linyouyi@hadoop01 hadoop-2.7.7]$ yarn rmadmin -refreshNodes 17/06/24 14:55:56 INFO client.RMProxy: Connecting to ResourceManager at hadoop02/172.16.2.103:8033
4. Check the Web browser, the state of the node is retired decommission in progress (decommissioning), a data block is being copied node to the other node
5. The node status is waiting for retirement decommissioned (all blocks have been copied completed), and stopping the node node Explorer. Note: If the number of copies to be decommissioned 3, serving node is less than or equal to 3, is not retired successful, you need to modify the number of copies
[linyouyi@hadoop04 hadoop-2.7.7]$ sbin/hadoop-daemon.sh stop datanode stopping datanode [linyouyi@hadoop04 hadoop-2.7.7]$ sbin/yarn-daemon.sh stop nodemanager
6. If the data is not balanced, you can use the command rebalance cluster
[linyouyi@hadoop01 hadoop-2.7.7]$ sbin/start-balancer.sh starting balancer, logging to /hadoop/module/hadoop-2.7.7/logs/hadoop-linyouyi-balancer-hadoop01.out Time Stamp Iteration# Bytes Already Moved Bytes Left To Move Bytes Being Moved
Note: Do not allow white and black lists also appear in the same host name