Hadoop distributed cluster deployment and installation
1 Hadoop distributed cluster deployment and installation
In hadoop2.0, it usually consists of two NameNodes, one in active state and the other in standby state. The Active NameNode provides external services, while the Standby NameNode does not provide external services, but only synchronizes the status of the activenamenode so that it can quickly switch if it fails.
Hadoop 2.0 officially provides two HDFS HA solutions, one is NFS and the other is QJM. Here we use simple QJM. In this solution, metadata information is synchronized between the active and standby NameNodes through a set of JournalNodes, and a piece of data is considered successful as long as it is successfully written to a majority of JournalNodes. Usually an odd number of JournalNodes
are configured. A zookeeper cluster is also configured here for ZKFC (DFSZKFailoverController) failover. When the ActiveNameNode hangs, it will automatically switch the Standby NameNode to the active state.
1.1 Working principle of Hadoop distributed cluster
1.2 Cluster planning
Host name IP Installation software Running process
ha1 192.168.137.171 Jdk, Scala, Zookeep, Hadoop, Spark NameNode, DataNode, JournalNode, QuorumPeerMain, DFSZKFailoverController, ResourceManager, NodeManager, Spark Worker
ha2 192.168. 137.172 Jdk, Scala, Zookeep, Hadoop, Spark NameNode, DataNode, JournalNode, QuorumPeerMain, DFSZKFailoverController, NodeManager, Spark Worker
ha3 192.168.137.173 Jdk, Scala, Zookeep, Hadoop, Spark DataNode, JournalNode, QuorumPeerMain, ResourceManager, NodeManager, Spark Master, Spark Worker
1.3 related software
1, jdk-8u131-linux-x64.tar.gz
2, hadoop-2.6.4.tar.gz
3, zookeeper-3.4.5.tar.gz
4, scala-2.10.4.tgz
5 , spark-1.6.1-bin-hadoop2.6.tgz
1.4 Modify the host name to configure the hosts file
Modify the host name
Modify the host name of the ha1 virtual machine:
execute the command: vi /etc/sysconfig/network
Modify HOSTNAME=ha1.ry600.com in the
same way, modify ha2 to ha2.ry600.com
, modify ha3 to ha3.ry600 .com
Configure the ha1 server and execute the command: vi /etc/hosts
127.0.0.1 localhost
192.168.137.171 ha1.ry600.com
192.168.137.172 ha2.ry600.com
192.168.137.173 ha3.ry600.com
Copy other server scp commands:
scp /etc/hosts 192.168.137.172:/etc/
scp /etc/hosts 192.168.137.173:/etc/
1.5 Configure ssh password-free login
Generate keys, execute the command: ssh-keygen -t rsa
Press 4 Enter, the key file is located in the ~/.ssh file
. Generate a pair of keys on ha1, copy the public key to other nodes, including yourself, and execute the command:
ssh-copy-id ha1.ry600.com
ssh-copy- id ha2.ry600.com
ssh-copy-id ha3.ry600.com
(Optional: You can continue to generate keys on ha2 and ha3 and copy them to other nodes)
1.6 Turn off the firewall
Turn off the command: service iptables stop
Permanently turn off the firewall: chkconfig iptables off
Run the two commands at the same time. After the operation is complete, check the firewall shutdown
status . zxvf jdk-8u131-linux-x64.tar.gz decompress: Hadoop: execute the command: tar -zxvf hadoop-2.6.4.tar.gz decompress Zookeeper: execute the command: tar -zxvf zookeeper-3.4.5.tar.gz decompress Scala: Execute the command: tar -zxvf scala-2.10.4.tgz Extract Spark: Execute the command: tar -zxvf spark-1.6.1-bin-hadoop2.6.tgz
1.8 Set the environment variable
ha1 to modify the profile file:
execute the command: vi /etc/profile
Add at the end of the file:
export JAVA_HOME=/hasoft/jdk1.8.0_131
export SCALA_HOME=/hasoft/scala-2.10.4
export HADOOP_HOME=/hasoft/ hadoop-2.6.4
export SPARK_HOME=/hasoft/spark-1.6.1-bin-hadoop2.6
export PATH=${JAVA_HOME}/bin:${SCALA_HOME}/bin:${HADOOP_HOME}/bin:${HADOOP_HOME} /sbin:${SPARK_HOME}/bin:${SPARK_HOME}/sbin:$PATH
1.9 Zookeeper cluster configuration:
1. ha1 modify the zoo.cfg file:
execute the command:
cd /hasoft/zookeeper-3.4.5/conf
cp zoo_sample.cfg zoo.cfg
modify the content, execute the command: vi zoo.cfg
tickTime=2000
initLimit= 10
syncLimit=5
clientPort=2181
dataDir=/hasoft/zookeeper-3.4.6/data
dataLogDir=/hasoft/zookeeper-3.4.6/logs
server.1=ha1.ry600.com :2888:3888
server.2=ha2. ry600.com:2888:3888
server.3 = ha3.ry600.com:2888:3888 2. Create a folder for
server identification configuration , command: cd /hasoft/zookeeper-3.4.5 Command: mkdir data command i (equivalent to create myid file, the content is the server ID: 1). 1.10 hadoop configuration file (ha1): 1. Modify the hadoop-env.sh configuration file: execute the command:
vi /hasoft/hadoop-2.6.4/etc/hadoop/hadoop-env.sh
Modified content:
export JAVA_HOME=/hasoft/jdk1.8.0_131
2. Switch the directory and
execute the command: cd /hasoft/hadoop-2.6.4/etc /hadoop
3. Modify the core-site.xml configuration file:
execute the command:
vi core-site.xml
Modify the content:
<configuration>
<!--Specify the URL address of the nameService1 node of hdfs -->
<property>
<name>fs.defaultFS</name>
<value>hdfs://ns1</value>
</property>
<!-- Hadoop runtime file storage path, if this directory does not exist, it needs to be formatted -->
<property>
<name>hadoop.tmp.dir</name>
<value>/hasoft/hadoop-2.6.4/tmp</value >
</property>
<!-- Zookeeper address -->
<property>
<name>ha.zookeeper.quorum</name>
<value>ha1.ry600.com:2181,ha2.ry600.com:2181,ha3. ry600.com:2181</value>
</property>
</configuration>
4. Modify the hdfs-site.xml configuration file:
execute the command:
vi hdfs-site.xml
Modify the content:
<configuration>
<!--Specify the nameservice of hdfs as ns1, which needs to be consistent with core-site.xml -->
<property>
<name>dfs.nameservices</name>
<value>ns1</value>
< /property>
<!-- There are two NameNodes under ns1, nn1, nn2 -->
<property>
<name>dfs.ha.namenodes.ns1</name>
<value>nn1,nn2</value>
< /property>
<!-- nn1's RPC address -->
<property>
<name>dfs.namenode.rpc-address.ns1.nn1</name>
<value>ha1.ry600.com:9000</value>
< /property>
<!-- http communication address of nn1 -->
<property>
<name>dfs.namenode.http-address.ns1.nn1</name>
<value>ha1.ry600.com:50070</value>
</property>
<!-- nn2的RPC通信地址 -->
<property>
<name>dfs.namenode.rpc-address.ns1.nn2</name>
<value>ha2.ry600.com:9000</value>
</property>
<!-- nn2的http通信地址 -->
<property>
<name>dfs.namenode.http-address.ns1.nn2</name>
<value>ha2.ry600.com:50070</value>
</property>
<!-- 指定NameNode的元数据在JournalNode上的存放位置 -->
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://ha1.ry600.com:8485;ha2.ry600.com:8485;ha3.ry600.com:8485/ns1</value>
</property>
<!-- Specify JournalNode to store on local disk Location of data -->
<property>
<name>dfs.journalnode.edits.dir</name>
<value>/hasoft/hadoop-2.6.4/journal</value>
</property>
<!-- start NameNode Failover automatic switching -->
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property>
<!-- Configuration failure automatic switching implementation -->
<property>
<name>dfs. client.failover.proxy.provider.ns1</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<!-- Configure the isolation mechanism method, multiple mechanisms are separated by newlines, that is, each mechanism temporarily uses a line- ->
<property>
<name>dfs.ha.fencing.methods</name>
<value>
sshfence
shell(/bin/true)
</value>
</property>
<!-- ssh free is required when using sshfence isolation mechanism Login -->
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/root/.ssh/id_rsa</value>
</property>
<!-- Configure sshfence isolation mechanism timeout -->
<property>
<name>dfs.ha.fencing.ssh.connect-timeout</name>
<value>30000</value>
</property>
</configuration>
5. Modify the mapred-site.xml configuration file:
execute the copy command: cp mapred-site.xml.template mapred-site.xml
modification file: vi mapred-site.xml
modification content:
<configuration>
<!-- Specify that Hadoop MapReduce runs in the YARN environment -->
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
6 , Modify the yarn-site.xml configuration file:
execute the command:
vi yarn-site.xml
Modify the content:
<configuration>
<!-- 开启RM高可靠 -->
<property>
<name>yarn.resourcemanager.ha.enabled</name>
<value>true</value>
</property>
<!-- 指定RM的cluster id -->
<property>
<name>yarn.resourcemanager.cluster-id</name>
<value>YarnCluster</value>
</property>
<!-- 指定RM的名字 -->
<property>
<name>yarn.resourcemanager.ha.rm-ids</name>
<value>rm1,rm2</value>
</property>
<!-- 分别指定RM的地址 -->
<property>
<name>yarn.resourcemanager.hostname.rm1</name>
<value>ha1.ry600.com</value>
</property>
<property>
<name>yarn.resourcemanager.hostname.rm2</name>
<value>ha3.ry600.com</value>
</property>
<property>
<name>yarn.resourcemanager.recovery.enabled</name>
<value>true</value>
</property>
<property>
<name>yarn.resourcemanager.store.class</name>
<value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
</property>
<!-- Specify the zk cluster address-->
<property>
<name>yarn.resourcemanager.zk-address</name>
<value>ha1.ry600.com:2181,ha2.ry600.com:2181,ha3.ry600 .com:2181</value>
</property>
<!-- NodeManager gets data: shuffle -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
7. Modify the slaves configuration file:
execute the command: vi slaves
The modified content is:
ha1.ry600.com
ha2.ry600.com
ha3.ry600.com
1.11 Spark configuration file (ha1)
1 Configure spark-env.sh
Execute the command: cd /hasoft/spark-1.6.1-bin-hadoop2.6/conf
Execute the command: cp spark-env.sh.template spark-env.sh
Execute the command: vi spark-env .sh
Add the following content to the spark-env.sh file:
export JAVA_HOME=/hasoft/jdk1.8.0_131
export SCALA_HOME=/hasoft/scala-2.10.4
export HADOOP_CONF_DIR=/hasoft/hadoop-2.6.4/etc/hadoop
2 Configure slaves
to execute the command: cp slaves.template slaves
to execute the command: vi slaves
The contents of the slaves file are modified as follows:
# A Spark Worker will be started on each of the machines listed below.
ha1.ry600.com
ha2.ry600.com
ha3.ry600.com
1.12 ha1 Copy programs and configuration files to other servers (ha2, ha3)
1 Copy files
Execute the command: scp -r /etc/profile ha2.ry600.com:/etc/Execute
the command: scp -r /etc/profile ha3.ry600 .com:/etc/
Execute command: scp -r /hasoft ha2.ry600.com:/Execute
command: scp -r /hasoft ha3.ry600.com:/
2 Modify the configuration file
Execute the command: vi /hasoft/zookeeper-3.4.6/data/myid
Change the myid content of ha2 to 2
and change the myid content of ha3 to 3
3 Refresh environment variables
Execute the command: source /etc/profile refresh environment variables
1.13 Start Zookeeper:
1 Start zookeeper on ha1, ha2, and ha3 respectively and
execute the command: /hasoft/zookeeper-3.4.6/bin/zkServer.sh start
2 After all machines are started, check the startup status.
Execute the command: /hasoft/zookeeper-3.4.6/bin/zkServer.sh status
1.14 Start journalnode: Start journalnode
on ha1, ha2, and ha3 respectively, note: the script called hadoop-daemon.sh:
execute the command respectively: /hasoft/hadoop-2.7.3/sbin/hadoop-daemon.sh start journalnode
1.15 Format the file system:
On the ha1 server: format the HDFS file system, execute the command:
hdfs namenode -format
Verification: The following information is displayed to indicate success:
INFO common.Storage: Storage directory/usr/local/software/hadoop_2.7.1/tmp/dfs/name has been successfully formatted.
The Hadoop runtime file storage path is: tmp directory, ha1 needs to be the same as ha2, copy (nameNode metadata):
execute the command: scp -r /hasoft/hadoop-2.7.3/tmp ha2.ry600.com:/hasoft/hadoop-2.7.3/
1.16 Format formatZK: You
only need to execute the name on ha1:
Execute the command: hdfs zkfc -formatZK
Verification:
Open the zookeeper client of the ha1 server to see if there is a hadoop-ha node
Execute the command: /hasoft/zookeeper-3.4.6 /bin/zkCli.sh
View node command: ls /
1.17 Start HDFS:
1. Only need to start HDFS on ha1,
execute the command: /hasoft/hadoop-2.7.3/sbin/start-dfs.sh to
verify that HDFS is started:
Java Jps: JDK provides a small tool to view the current process.
NameNode: It is the master server in Hadoop, managing the file system namespace and access to files stored in the cluster.
DFSZKFailoverController will periodically call proxy.monitorHealth() through the rpc to monitor the health of the NN.
1.18 Start YARN:
1. Just start YARN on ha1 and
execute the command: /hasoft/hadoop-2.7.3/sbin/start-yarn.sh
2. Verify that YARN is started:
Jps: JDK provides a small tool to view the current java process.
ResourceManager: Receives client task requests, receives and monitors resource status reports from NodeManager (NM), is responsible for resource allocation and scheduling, and starts and monitors ApplicationMaster (AM).
1.19 Start Spark
ha1 and execute the command: /hasoft/spark-2.2.0-bin-hadoop2.7/sbin/start-all.sh
1.20 Access the service page:
access address verification to start the service, access address:
HDFS management interface: http://192.168.137.171:50070/
YARN management interface: http://192.168.137.171:8088/
http://192.168.137.171:8080/SPARK management interface: :
1.21 Verify HDFS - HA master/slave switchover:
1. Verify whether the ha1 server NameNode and ha2 server NameNode data files can be shared:
Upload the file hosts file on ha1:
execute the command: hadoop fs -put /etc/hosts /
Kill the ha1 NameNode process, execute the command:
View process: jps
Kill -9 6068
Verification succeeded
2. Start the NameNode process on ha1, forcibly shut down the ha2 server, and verify whether ha1 can be activated:
Start the NameNode process of the ha1 server:
execute the command: /hasoft/hadoop-2.6.4/sbin/hadoop-daemon.sh start namenode
Shut down the ha2 service:
Active-standby switch comparison diagram:
1.22 Daily startup steps:
Start zookeeper on ha1, ha2, and ha3 respectively and
execute the command: /hasoft/zookeeper-3.4.5/bin/zkServer.sh start
Just start HDFS on ha1 or ha2,
execute the command: /hasoft/hadoop-2.6.4/sbin/start-dfs.sh
Just start YARN on ha1 or ha3 and
execute the command: /hasoft/hadoop-2.6.4/sbin/start-yarn.sh to
start the ResourceManager of another machine and
execute the command: /hasoft/hadoop-2.6.4/sbin/ yarn-daemon.sh start resourcemanager
Just start Spark on ha1 or ha2 or ha3:
execute the command: /hasoft/spark-1.6.1-bin-hadoop2.6/sbin/start-all.sh