hadoop configure rack awareness

Rack awareness script

Use rack awareness python3 write scripts, newspaper store to topology.py, give execute permissions

import sys
import os

DEFAULT_RACK="/default-rack"
HADOOP_CONF_DIR = os.environ['HADOOP_CONF_DIR']
FILE="topology.data"

host_rack={}
for line in open(os.path.join(HADOOP_CONF_DIR, FILE)):
    (host, rack) = line.split()
    host_rack[host] = rack
#获取命令行的输入
for host in sys.argv[1:]:
    if host in host_rack:
        print(host_rack[host])
    else:
        print(DEFAULT_RACK)

　　

topology.data document reads as follows:

[root@test ~ 18:29:00]#cat /data3/hadoop/hadoop/etc/hadoop/topology.data 
192.168.4.46  /dn1/rack
192.168.4.47  /dn2/rack

Execute scripts

[root@test script_py 18:33:38]#python topology.py 192.168.4.46 192.168.4.47 192.168.4.48
/dn1/rack
/dn2/rack
/default-rack

 

shell scripting, reported deposit to topology.sh, give execute permissions.

! # / bin / the bash 
# HADOOP_CONF = / tmp 
the while [$ # -gt 0]; 
do 
 nodeArg. 1 = $ 
# file contents input to the standard input stream 
 Exec <$ {} /topology.data the HADOOP_CONF_DIR 
 Result = "" 
 the while Read Line 
 do 
# each row is defined as an array of input 
 Ar = ($ line) 
 IF [ "$ {Ar [0]}" = "$ nodeArg"] || # [ "$ {Ar [. 1]}" = "$ nodeArg" ] 
     the then 
     Result = "$ {Ar [. 1]}" 
 Fi 
 DONE 
 Shift 
 IF [the -Z "$ Result"] 
     the then 
     echo "/ Rack-default" 
 the else 
     echo "$ Result" 
 Fi 
DONE

　　

Execute the script:

[root@test ~ 18:35:07]#bash topology.sh 192.168.4.46 192.168.4.47 192.168.4.48
/dn1/rack
/dn2/rack
/default-rack

 

Core-site.xml configuration file

<property>  
  <name>topology.script.file.name</name>  
  <value>${HADOOP_CONF_DIR}/topology.sh</value>  
</property>