Host configuration:
Host: Xiaohei T450
CPU: I5
Memory: 16G
System: Windows 10 Home Edition 64-bit
Virtual Machine:
Oracle VM VirtualBox:
VirtualBox-5.1.30.18389-Win.exe
System Preparation:
CentOS-7-x86_64-Minimal -1708.iso
CentOS-7-x86_64-DVD-1708.iso
Linux system installation:
CentOS-7-x86_64-Minimal-1708.iso #Minimal installation, in order to save system resources
Download the DVD version, the purpose is from the software source of yum, for the required software such as: vim ssh net-tools, you need to hang on the local source, install
CentOS-7-x86_64-DVD-1708.iso
One: System environment configuration before installing hadoop:
1> Mount the yum software source:
The first step: CD-ROM loading ISO file: CentOS-7-x86_64-DVD-1708.iso
The operation is as follows
Step 2: Mount the CentOS system
Linux external storage is generally mounted in the /mnt directory, here I first create a folder cdrom to accommodate this optical drive:
-->cd /mnt
-->mkdir cdromeThen mount the CD-ROM to this folder:
-->mount /dev/cdrom /mnt/cdrom #/dev/cdrom是光驱的默认路径
Step 3: Configure the yum package source (using the local source we mounted in the previous step)
/etc/yum.repos.d/ directory
delete
CentOS-Base.repo
CentOS-Debuginfo.repo
-->cd /etc/yum.repos.d
-->rm -f CentOS-Base.repo
-->rm -f CentOS-Debuginfo.repo
Edit CentOS-Media.repo, specify the source path as the mount source path, and enable the use of
-->vi CentOS-Media.repo
-->baseurl:file:///mnt/cdrom/ #指定本地资源挂载路径
enabled=1 #开启
2> Install the following software
vim #Powerful text editor
net-tools #View network status software, such as viewing ifconfig commands, etc.
ssh #Remote secure login or file upload and download
Tip: If you don't know which command is provided by which package, you can check it like this:
-->yum provides ifconfig（具体命令）
或者
-->yum whatprovides ifconfig（具体命令）------------------------------------------------------------
-->yum -y install vim
-->yum -y install net-tools
-->yum -y install openssh* #表示安装源里openssh开头的软件全部安装
3> Modify the Linux system host name:
-->vim /etc/hostname
-->hadoop-ha-01
4> Configure the Linux system host to access the Internet
(1) The network card is set to bridge
(2) Set network card parameters
-->cd /etc/sysconfig/network-scripts/
-->ls #查找ifcfg-eth*
-->vim ifcfg-eth3 #打开文件编辑
-->onboot=yes #启用该网卡
(3) Modify the gateway
-->vim /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=hadoop-ha-01 #配置永久主机名
GATEWAY=192.168.1.21 #这里设置网关，也就是那个虚拟网卡的ip
(4) Modify DNS
-->vim /etc/resolv.conf
nameserver 192.168.1.1 #增加一个域名服务器
(5) Configure the firewall #Here first close it directly
-->systemctl stop firewalld.service #停止firewall
-->systemctl disable firewalld.service #禁止firewall开机启动
(6) After the network is configured, restart the network
-->service network restart
然后重启Linux系统
-->reboot
--------------------------------------------------------------------------------
Two: After the above is ready, we start to build the cluster
hadoop-ha cluster planning diagram
| hostname | IP | software | running process |
| hadoop-ha-01 | 192.168.1.21 | JDK、HADOOP | NameNode 、 DFSZKFailoverController (zkfc) |
| hadoop-ha-02 | 192.168.1.22 | JDK、HADOOP | NameNode 、 DFSZKFailoverController (zkfc) |
| hadoop-ha-03 | 192.168.1.23 | JDK、HADOOP | ResourceManager |
| hadoop-ha-04 | 192.168.1.24 | JDK、HADOOP | ResourceManager |
| hadoop-ha-05 | 192.168.1.25 | JDK、HADOOP、Zookeeper | DataNode、NodeManager、JournalNode、QuorumPeerMain |
| hadoop-ha-06 | 192.168.1.26 | JDK、HADOOP、Zookeeper | DataNode、NodeManager、JournalNode、QuorumPeerMain |
| hadoop-ha-07 | 192.168.1.27 | JDK、HADOOP、Zookeeper | DataNode、NodeManager、JournalNode、QuorumPeerMain |
illustrate:
hdfs file high availability configuration master and slave NameNode hosts hadoop-ha-01, hadoop-ha-02
Yarn management system high availability configuration master and slave ResourceManager hosts hadoop-ha-03, hadoop-ha-04
Because the node manager of the yarn management system NodeManager itself manages the execution of the MR program on the DataNode of the hdfs file system, configure them together on the remaining three machines
According to the cluster design diagram, 7 systems are required, so according to the hadoop-ha-01 image, 7 systems are built under the virtual machine
Note: After each system is built, you need to modify the host name and IP such as the design diagram
1> I use hadoop-ha-01 to manage the cluster, so here I configure 01 to access all machines in the cluster without password
(1) Configure ssh password-free login
-->ssh-keygen #生成密钥
-->ls /root/.ssh #查看生成密钥文件
结果：
id_rsa #私钥
id_rsa.pub #公钥
-->ssh-copy-id -i /root/.ssh/id_rsa.pub [email protected] #把公钥拷贝到远端服务器
-->ssh-copy-id -i /root/.ssh/id_rsa.pub [email protected] #把公钥拷贝到远端服务器
-->ssh-copy-id -i /root/.ssh/id_rsa.pub [email protected] #把公钥拷贝到远端服务器
-->ssh-copy-id -i /root/.ssh/id_rsa.pub [email protected] #把公钥拷贝到远端服务器
-->ssh-copy-id -i /root/.ssh/id_rsa.pub [email protected] #把公钥拷贝到远端服务器
-->ssh-copy-id -i /root/.ssh/id_rsa.pub [email protected] #把公钥拷贝到远端服务器
-->ssh-copy-id -i /root/.ssh/id_rsa.pub [email protected] #把公钥拷贝到远端服务器
2> All of the above are very difficult to remember and write with ip access. Here I configure the relationship between ip and host name in the hosts file.
--vim /etc/hosts
192.168.1.21 hadoop-ha-01
192.168.1.22 hadoop-ha-02
192.168.1.23 hadoop-ha-03
192.168.1.24 hadoop-ha-04
192.168.1.25 hadoop-ha-05
192.168.1.26 hadoop-ha-06
192.168.1.27 hadoop-ha-07
然后发送到集群每个机器上
scp /etc/hosts [email protected]:/etc/host
scp /etc/hosts [email protected]:/etc/host
scp /etc/hosts [email protected]:/etc/host
scp /etc/hosts [email protected]:/etc/host
scp /etc/hosts [email protected]:/etc/host
scp /etc/hosts [email protected]:/etc/host
2> Install the cluster to use the software package
jdk-8u162-linux-x64.tar.gz
hadoop-2.6.5.tar.gz
zookeeper-3.3.6.tar.gz
I have downloaded the above packages on the host machine, how can I send them to the virtual machine? I used the FileZilla tool to do this.
I am going to install all the software in the /usr/app directory, so all the machines in the cluster will create a new app directory (this can be done in advance)
-->mkdir /usr/app
3> Decompress jdk and send it to all machines in the cluster
-->tar -zxvf jdk-8u162-linux-x64.tar.gz4> Unzip zookeeper-3.3.6.tar.gz on hadoop-ha-05
-->tar -zxvf zookeeper-3.3.6.tar.gzConfigure zookeeper:
-->cd /root/app/zookeeper-3.3.6/conf/
-->cp zoo_sample.cfg zoo.cfg
-->vim zoo.cfg
dataDir=/root/app/zookeeper-3.3.6/tmp
#在最后添加：
server.1=hadoop-ha-05:2888:3888
server.2=hadoop-ha-06:2888:3888
server.3=hadoop-ha-07:2888:3888
#保存退出
Then create a tmp folder
-->mkdir /root/app/zookeeper-3.3.6/tmp
#Create a myid file on the cluster according to the zzo.cfg configuration file server.*
-->echo 1 > /root/app/zookeeper-3.3.6/tmp/myid #这个文件名必须事myid
Copy the configured zookeeper to other nodes
-->scp -r /root/app/zookeeper-3.3.6/ hadoop-ha-06:/root/app/
-->scp -r /root/app/zookeeper-3.3.6/ hadoop-ha-07:/root/app/
注意：修改hdp-ha-06、hdp-ha-07对应/hadoop-ha-/zookeeper-3.3.6/tmp/myid内容
hdp-ha-06：
echo 2 > /root/app/zookeeper-3.3.6/tmp/myid
hdp-ha-07：
echo 3 > /root/app/zookeeper-3.3.6/tmp/myid
5>Install hadoop #first on hadoop-ha-01
解压：
-->tar -zxvf hadoop-2.6.5.tar.gz
#hadoop configuration hadoo-env.sh
export JAVA_HOME=/usr/app/jdk1.8.0_162
#hadoop deployment core-site.xml
<configuration>
<!-- 指定hdfs的nameservice为ns1 -->
<property>
<name>fs.defaultFS</name>
<value>hdfs://ns1/</value>
</property>
<!-- 指定hadoop临时目录 -->
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/app/hadoop-2.6.5/tmp</value>
</property>
<!-- 指定zookeeper地址 -->
<property>
<name>ha.zookeeper.quorum</name>
<value>hadoop-ha-05:2181,hadoop-ha-06:2181,hadoop-ha-07:2181</value>
</property>
</configuration>
#hadoop configuration hdfs-site.xml
<configuration>
<!--指定hdfs的nameservice为ns1，需要和core-site.xml中的保持一致 -->
<property>
<name>dfs.nameservices</name>
<value>ns1</value>
</property>
<!-- ns1下面有两个NameNode，分别是nn1，nn2 -->
<property>
<name>dfs.ha.namenodes.ns1</name>
<value>nn1,nn2</value>
</property>
<!-- nn1的RPC通信地址 -->
<property>
<name>dfs.namenode.rpc-address.ns1.nn1</name>
<value>hadoop-ha-01:9000</value>
</property>
<!-- nn1的http通信地址 -->
<property>
<name>dfs.namenode.http-address.ns1.nn1</name>
<value>hadoop-ha-01:50070</value>
</property>
<!-- nn2的RPC通信地址 -->
<property>
<name>dfs.namenode.rpc-address.ns1.nn2</name>
<value>hadoop-ha-02:9000</value>
</property>
<!-- nn2的http通信地址 -->
<property>
<name>dfs.namenode.http-address.ns1.nn2</name>
<value>hadoop-ha-02:50070</value>
</property>
<!-- 指定NameNode的元数据在JournalNode上的存放位置 -->
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://hadoop-ha-05:8485;hadoop-ha-06:8485;hadoop-ha-07:8485/ns1</value>
</property>
<!-- 指定JournalNode在本地磁盘存放数据的位置 -->
<property>
<name>dfs.journalnode.edits.dir</name>
<value>/usr/app/hadoop-2.6.5/journaldata</value>
</property>
<!-- 开启NameNode失败自动切换 -->
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property>
<!-- 配置失败自动切换实现方式 -->
<property>
<name>dfs.client.failover.proxy.provider.ns1</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<!-- 配置隔离机制方法，多个机制用换行分割，即每个机制暂用一行-->
<property>
<name>dfs.ha.fencing.methods</name>
<value>
sshfence
shell(/bin/true)
</value>
</property>
<!-- 使用sshfence隔离机制时需要ssh免登陆 -->
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/root/.ssh/id_rsa</value>
</property>
<!-- 配置sshfence隔离机制超时时间 -->
<property>
<name>dfs.ha.fencing.ssh.connect-timeout</name>
<value>30000</value>
</property>
</configuration>
#hadoop placement mapred-site.xml
<configuration>
<!-- 指定mr框架为yarn方式 -->
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
#hadoop placement yarn-site.xml
<configuration>
<!-- 开启RM高可用 -->
<property>
<name>yarn.resourcemanager.ha.enabled</name>
<value>true</value>
</property>
<!-- 指定RM的cluster id -->
<property>
<name>yarn.resourcemanager.cluster-id</name>
<value>yrc</value>
</property>
<!-- 指定RM的名字 -->
<property>
<name>yarn.resourcemanager.ha.rm-ids</name>
<value>rm1,rm2</value>
</property>
<!-- 分别指定RM的地址 -->
<property>
<name>yarn.resourcemanager.hostname.rm1</name>
<value>hadoop-ha-03</value>
</property>
<property>
<name>yarn.resourcemanager.hostname.rm2</name>
<value>hadoop-ha-04</value>
</property>
<!-- 指定zk集群地址 -->
<property>
<name>yarn.resourcemanager.zk-address</name>
<value>hadoop-ha-05:2181,hadoop-ha-06:2181,hadoop-ha-07:2181</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
#hadoop configuration slaves
Modify slaves (slaves is the location of the specified child node, because HDFS is to be started on hadoop-ha-01 and yarn is started on hadoop-ha-03, so the slaves file on hadoop-ha-01 specifies datanode The location of the slaves file on hadoop-ha-03 specifies the location of the nodemanager)
hadoop-ha-05
hadoop-ha-06
hadoop-ha-07
Finally, copy the hadoop installation directory to each machine in the cluster
Slaves theoretically need to be modified! and
Start hdfs on hadoop-ha-01, so you need to know the location of datanode, so put the node that starts datanode into the slaves file of hadoop-ha-01 machine
Start yarn on hadoop-ha-03, so you need to know the location of nodemanager, so put the node that starts nodemanager in the slaves file of hadoop-ha-03 machine
---------------- -------------------------------------------------
Three : Configure environment variables:
-->vim /etc/profile
#jdk的集群所有机器都需要配
export JAVA_HOME=/usr/app/jdk1.8.0_162
export JRE_HOME=$JAVA_HOME/jre
export CLASSPATH=$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH
#hadoop的集群所有机器都需要配
export HADOOP_HOME=/usr/app/hadoop-2.6.5
export PATH=$JAVA_HOME/bin:$JRE_HOME/bin$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH #在jdk基础上改
#zookeeper只在05/06/07上配置
export ZOOKEEPER_HOME=/usr/app/zookeeper-3.3.6
export PATH=$JAVA_HOME/bin:$JRE_HOME/bin$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$ZOOKEEPER_HOME/bin:$PATH #在jdk基础上改
Four: After the construction is completed, start the startup #Need to pay attention to the startup sequence
#启动zookeeper集群（分别在hadoop-ha-05、hadoop-ha-06、hadoop-ha-上启动zk）
zkServer.sh start
#查看状态：一个leader，两个follower
zkServer.sh status
#启动journalnode（分别在在hadoop-ha-05、hadoop-ha-06、hadoop-ha-07上执行）
hadoop-daemon.sh start journalnode
#运行jps命令检验，hdp-ha-05、hdp-ha-06、hdp-ha-07上多了JournalNode进程
#格式化HDFS
#在hadoop-ha-01上执行命令:
hdfs namenode -format
#格式化后会在根据core-site.xml中的hadoop.tmp.dir配置生成个文件，这里我配置的是/usr/app/hadoop-2.6.5/tmp，然后将/usr/app/hadoop-2.6.5/tmp拷贝到hadoop-ha-02的/usr/app/hadoop-2.6.5/下。
scp -r tmp/ hadoop-ha-02:/root/app/hadoop-2.6.5/
#格式化ZKFC(在hdp-ha-01上执行即可)
hdfs zkfc -formatZK
#启动HDFS(在hdp-ha-01上执行)
start-dfs.sh
#启动YARN
#(是在hdp-ha-03上执行start-yarn.sh，把namenode和resourcemanager分开是因为性能问题，因为他们都要占用大量资源，所以把他们分开了，他们分开了就要分别在不同的机器上启动)
start-yarn.sh