Install Hadoop2.6.2 under Ubuntu16.04 (stand-alone mode)

1. Create a hadoop group and a hadoop user under Ubuntu.
    Add a hadoop user group, and add a hadoop user to the group. When it comes to hadoop operations, we will use this user.



1. Create a hadoop user group
    sudo addgroup hadoop

2. Create a hadoop user   

    sudo adduser -ingroup hadoop hadoop

    After pressing Enter, you will be prompted to enter a new UNIX password, which is the password of the newly created user hadoop, just press Enter.

    If you do not enter the password, you will be prompted to enter the password again after pressing Enter, that is, the password cannot be empty.

    Finally, confirm whether the information is correct. If there is no problem, enter Y and press Enter. 

3. Add permissions for the hadoop user
     Enter: sudo gedit /etc/sudoers

     Enter, open the sudoers file

     and give the hadoop user the same permissions as the root user

2. Log in to the Ubuntu system with the newly added hadoop user

       su hadoop

3. Install ssh
sudo apt -get install openssh-server

After the installation is complete, start the service

sudo /etc/init.d/ssh start



to check whether the service is started correctly: ps -e | grep ss Set password-free login, generate private key and public key

ssh-keygen -t rsa -P ""



At this time, two files will be generated under /home/hadoop/.ssh: id_rsa and id_rsa.pub, the former is the private key and the latter is the public key.

Below we append the public key to authorized_keys, which holds the public key content of all users allowed to log in to the ssh client as the current user.

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

Log in to ssh

ssh localhost

to exit

exit



Fourth, install the Java environment to
     view the installation results, enter the command: java -version, the results are as follows, indicating that the installation is successful.

     java version "1.8.0_77"

    Java(TM) SE Runtime Environment (build 1.8.0_77-b03)

    Java HotSpot(TM) 64-Bit Server VM (build 25.77-b03, mixed mode)



5. Install hadoop2.6.2
    1. Official website Download http://mirror.bit.edu.cn/apache/hadoop/common/


    2. Install and


        decompress

        sudo tar xzf hadoop-2.6.2.tar.gz       

        If we want to install hadoop under /usr/local and

        copy it to /usr/local/, the folder is hadoop

        sudo mv hadoop-2.6.2 /usr/local/hadoop          



give users read and write permissions to this folder

        sudo chmod 777 /usr/local/hadoop



3. Configuration
     

        1) Configure ~/.bashrc   

Before configuring this file, you need to know the Java installation path, which is used to set the JAVA_HOME environment variable. You can use the following command line to view the installation path

    sudo gedit ~/.bashrc

   

    this The command will open the editing window for the file, append the following to the end of the file, save it, and close the editing window.

#HADOOP VARIABLES START

export JAVA_HOME=/usr/lib/jvm/jdk1.8.0_77

export HADOOP_INSTALL=/usr/local/hadoop

export PATH=$PATH:$HADOOP_INSTALL/bin

export PATH=$PATH:$HADOOP_INSTALL/sbin

export HADOOP_MAPRED_HOME= $HADOOP_INSTALL

export HADOOP_COMMON_HOME=$HADOOP_INSTALL

export HADOOP_HDFS_HOME=$HADOOP_INSTALL

export YARN_HOME=$HADOOP_INSTALL

export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native

export HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib"

#HADOOP VARIABLES END



Execute the following command to make the added environment variables take effect:

        source ~/.bashrc

2) Edit /usr/local/hadoop/etc/hadoop/hadoop-env.sh



        Execute the following command to open the file editing window

        sudo gedit /usr/local/hadoop/etc/hadoop/hadoop-env .sh

Find the JAVA_HOME variable, modify this variable as follows:

        export JAVA_HOME=/usr/lib/jvm/jdk1.8.0_77

6. WordCount test The


stand-alone mode installation is completed, and the following is to verify whether the installation is successful by executing the hadoop own instance WordCount

    /usr/local/ Create input folder   

mkdir input under hadoop path and



    copy README.txt to input   

cp README.txt input

    execute WordCount

    bin/hadoop jar share/hadoop/mapreduce/sources/hadoop-mapreduce-examples-2.6.2-sources.jar org.apache.hadoop.examples.WordCount input output 





execute cat output/*, check Character Statistics