hadoop a deployment from a master (1)

First, before installation instructions

Host IP: 192.168.132.128

Slave IP: 192.168.132.129

1. I put all of the installation package under / root / directory, you want to go to modify according to their own circumstances, must pay attention to this point

2. The installation package using the following

jdk-7u79-linux-x64.tar.gz，hadoop-2.8.0.tar.gz

3. Linux system uses centos7.0

4. If there is no text in the command stressed default perform the same command on both loom

Second, the installation step

1 Configure a free confidential login
execute command ssh-keygen
has been pressing the Enter key.
Host execute commands ssh-copy-id [email protected]
from the command execution machine ssh-copy-id [email protected]
in accordance with the requirements input yes, and password

2 Verify avoid dense Login
host execute commands ssh 192.168.132.129
normally do not need to enter a password to log in, and then verify the successful implementation of the exit command to exit telnet.
The same operation is performed again from the machine, execute commands from the machine ssh 192.168.132.128.

3 were added under / etc / hosts file The following two lines:

192.168.132.128 master
192.168.132.129 slaver

4 decompression jdk and Hadoop, and add it to environment variables

tar zxvf jdk-7u79-linux-x64.tar.gz
tar zxvf hadoop-2.8.0.tar.gz

Write the following code (in my path / root / down, you have to change your own path) in the / etc / profile:

export JAVA_HOME=/root/jdk1.7.0_79
export HADOOP_HOME=/root/hadoop-2.8.0
export PATH=$PATH:$JAVA_HOME/bin
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin

Run source /etc/profilethe environment variables to take effect

5 Create the necessary directories used to store data and temporary files (these directories will be used in the configuration file)

mkdir tmp dfs
mkdir dfs/data dfs/name

6 pairs Hadoop configuration file is modified
to perform the command cd /root/hadoop-2.8.0/etc/hadoop/
will hadoop-env.sh, yarn-env.sh, the value of JAVA_HOME mapred-env.sh file into
/root/jdk1.7.0_79
think the trouble can be manually modified using the following three commands:

sed -i '/^export JAVA_HOME=${JAVA_HOME}/ s/JAVA_HOME=${JAVA_HOME}/JAVA_HOME=\/root\/jdk1.7.0_79/g' hadoop-env.sh
sed -i '/^#.*export JAVA_HOME=.*/ s/^.*$/export JAVA_HOME=\/root\/jdk1.7.0_79/g' yarn-env.sh
sed -i '/^#.*export JAVA_HOME=.*/ s/^.*$/export JAVA_HOME=\/root\/jdk1.7.0_79/g' mapred-env.sh

Seven pairs of Hadoop core-site.xml configuration file to be modified, run
sed -i '/<.*configuration>/d' core-site.xml
the above command is to clean up the original configuration file, and then execute the following command to write the new configuration file (configuration file from the host machine no difference, all with IP host):

cat >> core-site.xml << EOF
<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://master:9000</value>
    </property>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>file:/root/tmp</value>
    </property>
    <property>
        <name>io.file.buffer.size</name>
        <value>131702</value>
    </property>
</configuration>
EOF

8 Profile of the Hadoop hdfs-site.xml, (where the difference between the host and slave configuration file slightly) Executive order:
sed -i '/<.*configuration>/d' hdfs-site.xml
The above command is the original configuration files to clean up, and then from the machine !! (if it is the host, and require the following configuration file slaver into a master, because the master and slave profile is slightly different, and execute the configuration file from the host machine with their own IP,) the following command to write the new configuration file:

cat >> hdfs-site.xml << EOF
<configuration>
    <property>
        <name>dfs.namenode.name.dir</name>
        <value>file:/root/dfs/name</value>
    </property>
    <property>
        <name>dfs.datanode.data.dir</name>
        <value>file:/root/dfs/data</value>
    </property>
    <property>
        <name>dfs.replication</name>
        <value>2</value>
    </property>
    <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>slaver:9001</value>
    </property>
    <property>
    <name>dfs.webhdfs.enabled</name>
    <value>true</value>
    </property>
</configuration>
EOF

9 pairs Hadoop profile mapred-site.xml modified
Run mv mapred-site.xml.template mapred-site.xml
Run sed -i '/<.*configuration>/d' mapred-site.xml
from the machine to execute a large segment command (if the host, into the slaver master, because the profiles are different from the host machine, the configuration file with their own IP):

cat >> mapred-site.xml << EOF
<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.address</name>
        <value>slaver:10020</value>
    </property>
    <property>
        <name>mapreduce.jobhistory.webapp.address</name>
        <value>slaver:19888</value>
    </property>
</configuration>
EOF

10 pairs of profile modification Hadoop yarn-site.xml
Run: sed -i '/<.*configuration>/d' yarn-site.xml
execute a large section of the command (the configuration file from the host machine no difference, both host computer):

cat >> yarn-site.xml << EOF
<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.auxservices.mapreduce.shuffle.class</name>
        <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>
    <property>
        <name>yarn.resourcemanager.address</name>
        <value>master:8032</value>
    </property>
    <property>
        <name>yarn.resourcemanager.scheduler.address</name>
        <value>master:8030</value>
    </property>
    <property>
        <name>yarn.resourcemanager.resource-tracker.address</name>
        <value>master:8031</value>
    </property>
    <property>
        <name>yarn.resourcemanager.admin.address</name>
        <value>master:8033</value>
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>master:8088</value>
    </property>
    <property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>4096</value>
    </property>
</configuration>
EOF

11 pairs Hadoop configuration file slavers to modify
the inside of the localhostdeletion
writeslaver

So far, the preliminary build Hadoop is completed, the next post will be brief and easy to use Hadoop initialization. This code has been tested many times, if there is a problem

Check the absolute path
Check whether the omission of some commands