Big Data Hadoop availability HA (Journal, ZooKeeper)

　　　　　　NN1 NN2 DN ZK ZKFC JNN
Node01 * * *
NODE02 * * * * *
NODE03 * * *
NODE04 * *

1, first of all let two key NameNode can avoid each other, the front part of node01 node02 has been able to avoid secret landing, so let node02 and node01 own free secret landing on the line

[root@node02 .ssh]# ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
[root@node02 .ssh]# cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
[root@node02 .ssh]# scp id_dsa.pub node01:`pwd`/node02.pub

[root@node01 .ssh]# cat ~/.ssh/node02.pub >> ~/.ssh/authorized_keys

2, modify the configuration file Hadoop, hdfs-site.xml

[root@node01 hadoop]# vi hdfs-site.xml
delete
    <property>
        <name>dfs.namenode.secondary.http-address</name>
        <value>node02:50090</value>
    </property>

Add to
    <property>
        <Name> dfs.nameservices </ name> // master node service name
        <value>mycluster</value>
    </property>
    <property>
        <name>dfs.ha.namenodes.mycluster</name>　　//namenode ID
        <value>nn1,nn2</value>
    </property>
    <property>
        <name>dfs.namenode.rpc-address.mycluster.nn1</name>　　//namenode地址
        <value>node01:8020</value>
    </property>
    <property>
        <name>dfs.namenode.rpc-address.mycluster.nn2</name>
        <value>node02:8020</value>
    </property>
    <property>
        <Name> dfs.namenode.http-address.mycluster.nn1 </ name> // namenode the http service address
        <value>node01:50070</value>
    </property>
    <property>
        <name>dfs.namenode.http-address.mycluster.nn2</name>
        <value>node02:50070</value>
    </property>

　　<property>
　　　　<name>dfs.namenode.shared.edits.dir</name>　　//JournalNode地址
　　　　<value>qjournal://node01:8485;node02:8485;node03:8485/mycluster</value>
　　</property>

　　<property>
　　　　<name>dfs.journalnode.edits.dir</name>　　　　//JournalNode的edits保存路径
　　　　<value>/var/ycyz/hadoop/journal</value>
　　</property>

　　<property>
　　　　<name>dfs.client.failover.proxy.provider.mycluster</name>　　//故障转移的代理模式
　　　　<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
　　</property>

　　<property>
　　　　<name>dfs.ha.fencing.methods</name>Fault isolation state namenode // 　　</ Property>
　　　　<value> sshfence </ value>

　　<property>
　　　　<name>dfs.ha.fencing.ssh.private-key-files</name>　　//私钥路径
　　　　<value>/root/.ssh/id_dsa</value>
　　</property>

3, modified core-site.xml

[root@node01 hadoop]# vi core-site.xml 
   <property>
    　　<name>fs.defaultFS</name>
　　　　<value>hdfs://mycluster</value>
　　</property>

4, the configuration zookeeper, provided Hadoop automatic failover in the configuration file, and configure the cluster address zookeeper

[root@node01 hadoop]# vi hdfs-site.xml
　　<property>
　　　　<name>dfs.ha.automatic-failover.enabled</name>
　　　　<value>true</value>
　　</property>

[root@node01 hadoop]# vi core-site.xml
　　<property>
　　　　<name>ha.zookeeper.quorum</name>
　　　　<value>node02:2181,node03:2181,node04:2181</value>
　　</property>

5, the hdfs-site.xml and core-site.xml Hadoop distributed to all other nodes

[root@node01 hadoop]# scp hdfs-site.xml  core-site.xml node02:`pwd`

6, mounting zookeeper3.4.6, unpack the archive and the other path

[root@node02 software]# tar xf zookeeper-3.4.6.tar.gz -C /opt/ycyz/

7, into the zookeeper conf directory, copy and zoo.cfg zoo_sample.cfg to configure dataDir, cluster nodes

[root@node02 conf]# cp zoo_sample.cfg zoo.cfg
[root @ node02 conf] # vi zoo.cfg
    dataDir=/var/ycyz/zookeeper

    Server. . 1 = amdha02: 2888 : 3888 // cluster corresponding to the node ID 
    Server. 2 = node03: 2888 : 3888 
    Server. . 3 = node04: 2888 : 3888

8, the installation files will be distributed to other zookeeper zookeeper node

[root@node02 ycyz]# scp -r zookeeper-3.4.6/ node03:`pwd`

9, zookeeper each node creates dataDir of directory

[root@node02 ycyz]# mkdir -p /var/ycyz/zookeeper

10, inform each corresponding to its own node id zookeeper

[root@node02 ycyz]# echo 1 > /var/ycyz/zookeeper/myid

[root@node03 ~]# echo 2 > /var/ycyz/zookeeper/myid

[root@node04 ~]# echo 3 > /var/ycyz/zookeeper/myid

11, zookeeper configuration environment variable, and distributed to other nodes

[root@node02 zookeeper-3.4.6]# vi /etc/profile
    export ZOOKEEPER_HOME=/opt/ycyz/zookeeper-3.4.6
    PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$ZOOKEEPER_HOME/bin

[root@node02 zookeeper-3.4.6]# scp /etc/profile node03:/etc/
[root@node02 zookeeper-3.4.6]# source /etc/profile

12, start zookeeper cluster

Just start the Hadoop cluster and execute commands on a different node 
must be performed manually zkServer zookeeper on all nodes. SH Start 
View node status zkServer.sh status

13, started all JournalNode

hadoop-daemon.sh start journalnode

14, namenode format, if no two namenode format, the format in which a first namenode.

After formatting, starting namenode formatted, the formatting information is then synchronized to the node namenode unformatted.

[amdha01 the root @ ~] # HDFS NameNode - the format formatted
[amdha01 the root @ ~] # Hadoop-daemon. SH Start Start node NameNode

[amdha02 the root @ ~] # HDFS NameNode -bootstrapStandby node synchronization unformatted

15, in the format zkfc namenode

[root@node01 ~]# hdfs zkfc -formatZK

16, start Hadoop cluster, make sure before starting zookeeper cluster has started

[root@node01 ~]# start-dfs.sh

Big Data Hadoop availability HA (Journal, ZooKeeper)

Guess you like