New increase Hadoop node:
To a running Hadoop addition of several new cluster Nodes
1. Deployment on the new node java / hadoop program, configure the environment variables
2. increase the user on the new node from the master copy of the id_rsa.pub and configured authorized_keys
3. Set a new node / etc / the hosts , and as long as the machine master of host to
/etc/hosts
127.0.0.1 localhost
10.10.10.101 node1
10.10.10.102 node2
10.10.10.103 node3
10.10.10.104 node4
4. The establishment of a new node on relevant directories, and modify the owner, the contents in Node4 ~ / tmp directory under are deleted, otherwise it will be considered a node has been created.
5. master 's slaves file nodes increases relative to the (* Note: Only above configuration can NameNode)
/etc/masters
node1
/etc/slaves
node2
node3
node4
6. start on a new node datanode and tasktracker
localadmin@node4:~/hadoop-0.20.2/bin$ hadoop-daemon.sh start datanode
localadmin@node4:~/hadoop-0.20.2/bin$ hadoop-daemon.sh start tasktracker
Use 'Jps' checked.
In hdfs-site.xml set to increase in the balance of the bandwidth, the default is only 1M :
<property>
<name>dfs.balance.bandwidthPerSec</name>
<value>10485760</value>
<description>
Specifies the maximum bandwidth that each datanode can utilize for the balancing purpose in term of the number of bytes per second.
</description>
</property>
Run the following command:
hadoop/bin/start-balancer.sh -threshold 3
Balanced 10 nodes, mobile 400G data, it took me about 3 Ge Xiaoshi
The cluster is balanced. Exiting…Balancing took 2.9950980555555557 hours
Delete Hadoop node:
1. In hadoop the conf generate a directory excludes file / etc / NN-List-excluded , needs to write remove nodes
A line node , such as:
node4
<property>
<name>dfs.hosts.exclude</name>
<value>/etc/nn-excluded-list</value>
</property>
3. Copy the file to each node in the cluster
4. execution hadoop dfsadmin -refreshNodes command, it will in the background Block moving block
Removed from the Nodes move on to other Nodes above
The following 2 See ways Decommission state:
hadoop dfsadmin -report
http://10.10.10.101:50070/dfsnodelist.jsp
Executing Decommission , it will be displayed:
Decommission Status : Decommission in progress
After the execution is completed, it will show:
Decommission Status : DecommissionedReproduced in: https: //www.cnblogs.com/licheng/archive/2011/11/10/2244804.html