Hadoop cluster entire build process, including

Preparatory
Zookeeper install and configure the environment
Compile and install hadoop start
HDFS installation and management namenode dataname hard disk cluster resource management
Installation Boot yarn build MapReduce cpu and memory resources management

01 preparation:

1. Deployment environment

VMware15
CentOS7
jdk8

First, start the virtual machine, one Taiwan centos7, configuration, Huawei cloud yum source

[root@localhost ~]# cp -a /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.bak
[root@localhost ~]# wget -O /etc/yum.repos.d/CentOS-Base.repo https://repo.huaweicloud.com/repository/conf/CentOS-7-reg.repo
[root@localhost ~]# yum clean all
[root@localhost ~]# yum makecache
[root@localhost ~]# yum update -y

Then install some commonly used software mess

[root@localhost ~]# yum install -y openssh-server vim gcc gcc-c++ glibc-headers bzip2-devel lzo-devel curl wget openssh-clients zlib-devel autoconf automake cmake libtool openssl-devel fuse-devel snappy-devel telnet unzip zip net-tools.x86_64 firewalld systemd

2. Turn off the virtual machine's firewall and SELinux settings

[root@localhost ~]# firewall-cmd --state
[root@localhost ~]# systemctl stop firewalld.service
[root@localhost ~]# systemctl disable firewalld.service
[root@localhost ~]# systemctl is-enabled firewalld.service

[root@localhost ~]# /usr/sbin/sestatus -v  查看selinux的状态
[root@localhost ~]# vim /etc/selinux/config
#修改状态为关闭
SELINUX=disabled
[root@localhost ~]# reboot

3. Install and configure the environment variables jdk8

Download http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

[root@localhost ~]# rpm -ivh jdk-8u144-linux-x64.rpm
[root@localhost ~]# vim /etc/profile
#修改环境变量,在文件末尾添加如下
export JAVA_HOME=/usr/java/jdk1.8.0_144
export JRE_HOME=$JAVA_HOME/jre
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

After modifying the user corresponds to only this session to take effect, want to permanently take effect globally, we must

[root@localhost ~]# source /etc/profile

4. Install ntpdate service, each virtual machine is suspended after re-enabling convenient synchronization time

[root@localhost ~]# yum install -y ntp-tools
[root@localhost ~]# ntpdate ntp1.aliyun.com

5. Create hadoop users and groups and add the wheel group

[root@localhost ~]# useradd hadoop
[root@localhost ~]# passwd hadoop

Only allows users within the wheel group can su - command to log in as root root, improve security

[root@localhost ~]# sed -i 's/#auth\t\trequired\tpam_wheel.so/auth\t\trequired\tpam_wheel.so/g' '/etc/pam.d/su'
[root@localhost ~]# cp /etc/login.defs /etc/login.defs_bak
[root@localhost ~]# echo "SU_WHEEL_ONLY yes" >> /etc/login.defs

Add hadoop user into the wheel group

[root@localhost ~]# gpasswd -a hadoop wheel
[root@localhost ~]# cat /etc/group | grep wheel     查看hadoop有没有加入到wheel组

View user group configuration

6. Configure virtual machine hosts file

[root@localhost ~]# vim /etc/hosts
192.168.10.3 nn1.hadoop   #这个是本机ip，主机名稍后一起配置
192.168.10.4 nn2.hadoop
192.168.10.5 s1.hadoop
192.168.10.6 s2.hadoop
192.168.10.7 s3.hadoop

7. The virtual machine function clone 4 clone additional virtual machines

Upon completion _ were to change every host name _ and configure a static ip, within the above requirements and is consistent and corresponds hosts file

[root@localhost ~]# hostnamectl set-hostname nn1.hadoop
[root@localhost ~]# vim /etc/sysconfig/network-scripts/ifcfg-ens33
TYPE="Ethernet"
PROXY_METHOD="none"
BROWSER_ONLY="no"
BOOTPROTO="static"      #这里修改为static
IPADDR="192.168.10.3"   #这里添加为你的每台虚拟机对应的ip
NETMASK="255.255.255.0" #添加
GATEWAY="192.168.10.2"  #添加为你虚拟机内的网关
DNS="192.168.10.2"      #添加
NM_CONTROLLED="no"      #添加，必然改完文件自动生效，可能直接网络就挂掉了
DEFROUTE="yes"
IPV4_FAILURE_FATAL="no"
IPV6INIT="yes"
IPV6_AUTOCONF="yes"
IPV6_DEFROUTE="yes"
IPV6_FAILURE_FATAL="no"
IPV6_ADDR_GEN_MODE="stable-privacy"
NAME="ens33"
UUID="49f05112-b80b-45c2-a3ec-d64c76ed2d9b"
DEVICE="ens33"
ONBOOT="yes"

[root@localhost ~]# systemctl stop NetworkManager.service   停止网络管理服务
[root@localhost ~]# systemctl disable NetworkManager.service 开机禁止自启动
[root@localhost ~]# systemctl restart network.service  重启网络服务

At this point we should have five virtual machines are configured according to the following correspondence good ip and host name, five are set up hosts file

192.168.10.3 nn1.hadoop
192.168.10.4 nn2.hadoop
192.168.10.5 s1.hadoop
192.168.10.6 s2.hadoop
192.168.10.7 s3.hadoop

Then the firewall, selinux have been closed, jdk8 are properly installed and configured environment variables are properly built hadoop user group and add them into the wheel group.

8. The five machines arranged densely Free each log ssh

Top of the operations are carried out under the root user, virtually all operations performed after the user switched to hadoop now.

[root@nn1 ~]# su - hadoop     注意这里的“-”，意味着用户和环境变量同时切换
[hadoop@nn1 ~]$    这时候代表进入到了hadoop用户，还有#和$分别代表root用户和普通用户的身份区别

He began to build a free ssh secret

The idea is to first create their own key on each machine separately, and finally summarized these key.pub to ~ / .ssh / authorized_keys file and then distributed to all the machines together, this time on five machines to achieve a mutually-free secret ssh access.

[hadoop@nn1 ~]$ pwd 查看当前路径，确保在hadoop用户的home下
/home/hadoop 
[hadoop@nn1 ~]$ mkdir .ssh
[hadoop@nn1 ~]$ chmod 700 ./.ssh
[hadoop@nn1 ~]$ ll -a
drwx------  2 hadoop hadoop 132 7月  16 22:13 .ssh

[hadoop@nn1 ~]$ ssh-keygen -t rsa   创建key文件

At this time it completed the setup nn1 machine (nn1 as the main operation of the machine our future). According to the procedure above the remaining four machines are running again, and then each of the other machines ./ssh/id_rsa.pub 4 Rename (other than to avoid duplication and replacement), and then sent to the nn1 ./ssh/ under

[hadoop@nn2 ~]$ scp ~/.ssh/id_rsa.pub [email protected] ~/.ssh/id_rsa.pubnn2

This time should nn1 of ~ / .ssh / under there including himself five pub file (not the same name), and then put them all _ added _ to below the file

[hadoop@nn1 ~]$ touch authorized_keys
[hadoop@nn1 ~]$ chmod 600 authorized_keys
[hadoop@nn1 ~]$ cat ./ssh/id_rsa.pub >> authorized_keys
[hadoop@nn1 ~]$ cat ./ssh/id_rsa.pubnn2 >> authorized_keys
[hadoop@nn1 ~]$ cat ./ssh/id_rsa.pubs1 >> authorized_keys
…………

And then finally put this batch file is sent to the remaining four machines (forgot to write a batch script, so use scp command in order to send it)

So far five of the machine configuration ssh-free access each other tight end, we can test separately (slightly).

9. Batch Scripting

Because there are five machines ah, many operations have to move together, so it is necessary to perform batch script.

#文件名：ips
"nn1.hadoop" "nn2.hadoop" "s1.hadoop" "s2.hadoop" "s3.hadoop"

#!/bin/bash
#文件名：ssh_all.sh
RUN_HOME=$(cd "$(dirname "$0")"; echo "${PWD}")

NOW_LIST=(`cat ${RUN_HOME}/ips`)

SSH_USER="hadoop"
for i in ${NOW_LIST[@]}; do
    f_cmd="ssh $SSH_USER@$i \"$*\""
    echo $f_cmd
    if eval $f_cmd; then
        echo "OK"
    else 
        echo "FAIL"
    fi
done

#!/bin/bash
#文件名：ssh_root.sh
RUN_HOME=$(cd "$(dirname "$0")"; echo "${PWD}")

NOW_LIST=(`cat ${RUN_HOME}/ips`)

SSH_USER="hadoop"
for i in ${NOW_LIST[@]}; do
    f_cmd="ssh $SSH_USER@i ~/exe.sh \"$*\""
    echo $f_cmd
    if eval $f_cmd; then
        echo "OK"
    else 
        echo "FAIL"
    fi
done

#文件名exe.sh
cmd=$*

su - <<EOF
$cmd

EOF

#!/bin/bash
RUN_HOME=$(cd "(dirname "$0")"; echo "${PWD}")

NOW_LIST=(`cat ${UN_HOME}/ips`)

SSH_USER="hadoop"
for i in ${NOW_LIST[@]}; do
    f_cmd="scp $1 $SSH_USER@i:$2"
    echo $f_cmd
    if eval $f_cmd; then
        echo "ok"
    else
        echo "FAIL"
    fi
done

Preparatory work completed, the next to start the installation configuration zookeeper

Hadoop Cluster Setup -01 preparatory