hadoop high availability cluster configuration

hadoop2.0 has released a stable version, adding many features, such as HDFS HA, YARN and so on. The latest hadoop-2.4.1 added YARN HA

Note: The installation package of hadoop-2.4.1 provided by apache is compiled on a 32-bit operating system, because hadoop depends on some C++ native libraries,

So if you install hadoop-2.4.1 on a 64-bit operation, you need to recompile on the 64 operating system

(It is recommended to install a 32-bit system for the first time. I also uploaded the compiled 64-bit system to the group sharing. If you are interested, you can compile it yourself)

I won’t go into details about the preparatory work, it’s all introduced in the class

1. Modify the Linux hostname

2. Modify IP

3. Modify the mapping between hostname and IP

######Note######If your company rents servers or uses cloud hosts (such as Huawei hosts, Alibaba Cloud hosts, etc.)

What needs to be configured in /etc/hosts is the mapping relationship between the intranet IP address and the host name

4. Turn off the firewall

5. ssh free login

6. Install JDK, configure environment variables, etc.

Cluster planning:

The process running the software installed by the hostname IP

weekend01192.168.1.201jdk、hadoopNameNode、DFSZKFailoverController(zkfc)

weekend02192.168.1.202jdk、hadoopNameNode、DFSZKFailoverController(zkfc)

weekend03192.168.1.203jdk、hadoopResourceManager

weekend04192.168.1.204jdk、hadoopResourceManager

weekend05192.168.1.205jdk、hadoop、zookeeperDataNode、NodeManager、JournalNode、QuorumPeerMain

weekend06192.168.1.206jdk、hadoop、zookeeperDataNode、NodeManager、JournalNode、QuorumPeerMain

weekend07192.168.1.207jdk、hadoop、zookeeperDataNode、NodeManager、JournalNode、QuorumPeerMain

illustrate:

1. In hadoop2.0, it usually consists of two NameNodes, one is in the active state and the other is in the standby state. The Active NameNode provides external services, while the Standby NameNode does not provide external services, but only synchronizes the status of the active namenode so that it can quickly switch if it fails.

Hadoop 2.0 officially provides two HDFS HA solutions, one is NFS and the other is QJM. Here we use simple QJM. In this solution, metadata information is synchronized between the active and standby NameNodes through a set of JournalNodes, and a piece of data is considered successful as long as it is successfully written to a majority of JournalNodes. Usually an odd number of JournalNodes are configured

A zookeeper cluster is also configured here for ZKFC (DFSZKFailoverController) failover. When the Active NameNode hangs up, it will automatically switch the Standby NameNode to the standby state

2. There is still a problem in hadoop-2.2.0, that is, there is only one ResourceManager, there is a single point of failure, hadoop-2.4.1 solves this problem, there are two ResourceManagers, one is Active, the other is Standby, and the status is carried out by zookeeper coordination

installation steps:

1. Install and configure the zooekeeper cluster (on weekend05)

1.1 Decompression

tar -zxvf zookeeper-3.4.5.tar.gz -C /weekend/

1.2 Modify the configuration

cd /weekend/zookeeper-3.4.5/conf/

cp zoo_sample.cfg zoo.cfg

came zoo.cfg

修改：dataDir=/weekend/zookeeper-3.4.5/tmp

Add at the end:

server.1=weekend05:2888:3888

server.2=weekend06:2888:3888

server.3=weekend07:2888:3888

save and exit

Then create a tmp folder

mkdir /weekend/zookeeper-3.4.5/tmp

Create another empty file

touch /weekend/zookeeper-3.4.5/tmp/myid

Finally write the ID to the file

echo 1 > /weekend/zookeeper-3.4.5/tmp/myid

1.3 Copy the configured zookeeper to other nodes (first create a weekend directory in the root directory of weekend06 and weekend07: mkdir /weekend)

scp -r /weekend/zookeeper-3.4.5/ weekend06:/weekend/

scp -r /weekend/zookeeper-3.4.5/ weekend07:/weekend/

Note: Modify the content of weekend06 and weekend07 corresponding to /weekend/zookeeper-3.4.5/tmp/myid

weekend06 ：

echo 2 > /weekend/zookeeper-3.4.5/tmp/myid

weekend07 ：

echo 3 > /weekend/zookeeper-3.4.5/tmp/myid

2. Install and configure hadoop cluster (operate on weekend01)

2.1 Decompression

tar -zxvf hadoop-2.4.1.tar.gz -C /weekend/

2.2 Configure HDFS (all configuration files of hadoop2.0 are in the $HADOOP_HOME/etc/hadoop directory)

#Add hadoop to environment variables

vim /etc/profile

export JAVA_HOME=/usr/java/jdk1.7.0_55

export HADOOP_HOME=/weekend/hadoop-2.4.1

export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin

#hadoop2.0 configuration files are all under $HADOOP_HOME/etc/hadoop

cd /home/hadoop/app/hadoop-2.4.1/etc/hadoop

2.2.1 Modify hadoo-env.sh

export JAVA_HOME=/home/hadoop/app/jdk1.7.0_55

2.2.2 Modify core-site.xml

<name>fs.defaultFS</name>

</property>

<name>hadoop.tmp.dir</name>

<value>/home/hadoop/app/hadoop-2.4.1/tmp</value>

</property>

<name>ha.zookeeper.quorum</name>

<value>weekend05:2181,weekend06:2181,weekend07:2181</value>

</property>

</configuration>

2.2.3 Modify hdfs-site.xml

<name>dfs.nameservices</name>

</property>

<name>dfs.ha.namenodes.ns1</name>

</property>

<name>dfs.namenode.rpc-address.ns1.nn1</name>

<value>weekend01:9000</value>

</property>

<name>dfs.namenode.http-address.ns1.nn1</name>

<value>weekend01:50070</value>

</property>

<name>dfs.namenode.rpc-address.ns1.nn2</name>

<value>weekend02:9000</value>

</property>

<name>dfs.namenode.http-address.ns1.nn2</name>

<value>weekend02:50070</value>

</property>

<name>dfs.namenode.shared.edits.dir</name>

<value>qjournal://weekend05:8485;weekend06:8485;weekend07:8485/ns1</value>

</property>

<name>dfs.journalnode.edits.dir</name>

<value>/home/hadoop/app/hadoop-2.4.1/journaldata</value>

</property>

<name>dfs.ha.automatic-failover.enabled</name>

</property>

<name>dfs.client.failover.proxy.provider.ns1</name>

<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>

</property>

<name>dfs.ha.fencing.methods</name>

<value>

sshfence

shell(/bin/true)

</value>

</property>

<name>dfs.ha.fencing.ssh.private-key-files</name>

<value>/home/hadoop/.ssh/id_rsa</value>

</property>

<name>dfs.ha.fencing.ssh.connect-timeout</name>

</property>

</configuration>

2.2.4 Modify mapred-site.xml

<name>mapreduce.framework.name</name>

</property>

</configuration>

2.2.5 Modify yarn-site.xml

<name>yarn.resourcemanager.ha.enabled</name>

</property>

<name>yarn.resourcemanager.cluster-id</name>

</property>

<name>yarn.resourcemanager.ha.rm-ids</name>

</property>

<name>yarn.resourcemanager.hostname.rm1</name>

<value>weekend03</value>

</property>

<name>yarn.resourcemanager.hostname.rm2</name>

<value>weekend04</value>

</property>

<name>yarn.resourcemanager.zk-address</name>

<value>weekend05:2181,weekend06:2181,weekend07:2181</value>

</property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>

</configuration>

2.2.6 Modify slaves (slaves is the location of the specified child node, because HDFS is to be started on weekend01 and yarn is started on weekend03, so the slaves file on weekend01 specifies the location of the datanode, and the slaves file on weekend03 specifies the nodemanager. Location)

weekend05

weekend06

weekend07

2.2.7 Configure password-free login

#First, configure password-free login from weekend01 to weekend02, weekend03, weekend04, weekend05, weekend06, and weekend07

#produce a pair of keys on weekend01

ssh-keygen -t rsa

#Copy the public key to other nodes, including yourself

ssh-coyp-id weekend01

ssh-coyp-id weekend02

ssh-coyp-id weekend03

ssh-coyp-id weekend04

ssh-coyp-id weekend05

ssh-coyp-id weekend06

ssh-coyp-id weekend07

#Configure password-free login from weekend03 to weekend04, weekend05, weekend06, and weekend07

#produce a pair of keys on weekend03

ssh-keygen -t rsa

#Copy the public key to other nodes

ssh-coyp-id weekend04

ssh-coyp-id weekend05

ssh-coyp-id weekend06

ssh-coyp-id weekend07

#Note: SSH password-free login should be configured between two namenodes, don't forget to configure weekend02 to weekend01 login-free

Produce a pair of keys on weekend02

ssh-keygen -t rsa

ssh-coyp-id -i weekend01

2.4 Copy the configured hadoop to other nodes

scp -r /weekend/ weekend02:/

scp -r /weekend/ weekend03:/

scp -r /weekend/hadoop-2.4.1/ hadoop@weekend04:/weekend/

scp -r /weekend/hadoop-2.4.1/ hadoop@weekend05:/weekend/

scp -r /weekend/hadoop-2.4.1/ hadoop@weekend06:/weekend/

scp -r /weekend/hadoop-2.4.1/ hadoop@weekend07:/weekend/

### NOTE: Strictly follow the steps below

2.5 Start the zookeeper cluster (start zk on weekend05, weekend06, and weekend07 respectively)

cd /home/hadoop/app/zookeeper-3.4.5/bin/

./zkServer.sh start

#View status: one leader, two followers

./zkServer.sh status

2.6 Start journalnode (execute on weekend05, weekend06, and weekend07 respectively)

cd /home/hadoop/app/hadoop-2.4.1

sbin/hadoop-daemon.sh start journalnode

#Run the jps command to check that there are more JournalNode processes on weekend05, weekend06, and weekend07

2.7 Format HDFS

#Execute the command on weekend01:

hdfs purpose-format

#After formatting, a file will be generated according to the hadoop.tmp.dir configuration in core-site.xml. Here I configured /weekend/hadoop-2.4.1/tmp, and then /weekend/hadoop-2.4.1 /tmp is copied to /weekend/hadoop-2.4.1/ of weekend02.

scp -r tmp/ weekend02:/home/hadoop/app/hadoop-2.4.1/

##This is also possible, it is recommended to hdfs namenode -bootstrapStandby

2.8 Format ZKFC (just execute it on weekend01)

hdfs zkfc -formatZK

2.9 Start HDFS (execute on weekend01)

sbin/start-dfs.sh

2.10 Start YARN (######Note #####: start-yarn.sh is executed on weekend03, the namenode and resourcemanager are separated because of performance problems, because they both take up a lot of resources, so they are separated , they should be started on different machines when they are separated)

sbin/start-yarn.sh

At this point, hadoop-2.4.1 is configured, and you can count browser access:

http://192.168.1.201:50070

NameNode 'weekend01:9000' (active)

http://192.168.1.202:50070

NameNode 'weekend02:9000' (standby)

Verify HDFS HA

First upload a file to hdfs

hadoop fs -put /etc/profile /profile

hadoop fs -ls /

Then kill the active NameNode

kill -9 <pid of NN>

Access via browser: http://192.168.1.202:50070

NameNode 'weekend02:9000' (active)

At this time, the NameNode on weekend02 becomes active

While executing the command:

hadoop fs -ls /

-rw-r--r-- 3 root supergroup 1926 2014-02-06 15:36 /profile

The file you just uploaded still exists! ! !

Manually start the hung NameNode

sbin/hadoop-daemon.sh start namenode

Access via browser: http://192.168.1.201:50070

NameNode 'weekend01:9000' (standby)

Verify YARN:

Run the WordCount program in the demo provided by hadoop:

hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.4.1.jar wordcount /profile /out

OK, you're done! ! !

Some instructions to test the working state of the cluster:

bin/hdfs dfsadmin -report View the status information of each node of hdfs

bin/hdfs haadmin -getServiceState nn1 Get the HA state of a namenode node

sbin/hadoop-daemon.sh start namenode Start a namenode process alone

./hadoop-daemon.sh start zkfc Start a zkfc process alone

yarn-daemon.sh start resourcemanager starts the yarn node alone

If there are only 3 hosts, you can deploy and install according to the following plan

weekend01zookeeper journalnode namenode zkfc resourcemanager datanode

weekend02zookeeper journalnode namenode zkfc resourcemanager datanode

weekend03zookeeper journalnode datanode

hadoop high availability cluster configuration

Guess you like