1.HDFS-HA Overview
1) The so-called HA (high available), namely high availability (7 * 24 hours uninterrupted service).
2) The key strategy is to achieve high availability eliminates single points of failure . Strictly speaking HA HA mechanism should be divided into various components: the HDFS the HA and the Y ARN of HA .
Before 3) Hadoop2.0, single point of failure (SPOF) in HDFS cluster NameNode.
4) NameNode major impact HDFS cluster in two ways
- NameNode machine accidents, such as downtime, the cluster will not be available until the administrator restarted
- NameNode machines need to be upgraded, including software, hardware upgrades, this time the cluster will not be able to use
By configuring HDFS HA Active Standby two nameNodes / implemented in the cluster thermal NameNode Prepared to solve the above problems . If a fault occurs, such as a machine or the machine crashes need to upgrade and maintenance, then the switch can be quickly NameNode by this way to another machine.
(By double namenode single point of failure)
2.HDFS-HA mechanism
2.1 HDFS-HA work points
1) metadata management approach requires changes:
- Memory, keep a journal metadata;
- Edits log only namenode node Active state can do write operations;
- Namenode two edits can be read;
- Share of edits in a shared storage management (qjournal and NFS achieve two mainstream);
2) the need for a status management module (zkfailover)
- It implements a zkfailover, where each node resident in namenode
- Each zkfailover responsible for monitoring where their namenode node using the state identification zk
- When the need for the state to switch from zkfailover responsible for switching, the need to prevent brain split phenomenon when switching.
3) must be guaranteed between the two the NameNode SSH login without a password .
4) Isolation (Fence), ie the same time there is only one NameNode of external service
2.2 HDFS-HA automatic failover mechanism
Use hdfs haadmin -failover command can fail manually, in this mode, even if the active NameNode has failed, the system will not automatically transferred from active to standby NameNode NameNode, learn how to configure the following deployment HA automatic failover. Automatic failover added two new components to HDFS deployment: ZooKeeper and ZKFailoverController (ZKFC) process. (ZooKeeper coordination is to maintain a small amount of data, notify the client to change and monitor the client's fault service availability of these data.)
Automatic transfer depends on the HA failure following functions ZooKeeper of :
1 ) fault detection: the cluster each NameNode maintained in ZooKeeper in a lasting conversation , if the machine crashes, ZooKeeper the session is terminated , notice another ZooKeeper NameNode need to trigger a failover.
2 ) active NameNode selection: the ZooKeeper provides a simple mechanism for selecting a single node in the active state. If the current active duty NameNode crashes, another node may obtain a special exclusive lock from ZooKeeper to indicate that it should become active NameNode.
ZKFC another new component in automatic failover, is ZooKeeper client , also monitor and manage the state of NameNode. Each run NameNode host also runs a ZKFC process, ZKFC responsible for:
1 ) Health Monitoring: ZKFC use a health check command NameNode in the same host periodically with ping, as long as the state of health NameNode not reply, ZKFC think that the node is healthy. If the node crashes, freezes or enters an unhealthy state, health monitor identifies the node for non-health.
2 ) ZooKeeper session management: when the local NameNode is healthy, ZKFC to keep an open ZooKeeper in the session. If the local NameNode in the active state, ZKFC also maintained a special znode lock which uses ZooKeeper support for transient nodes, if a session is terminated, the node will lock automatically deleted.
3 ) based on ZooKeeper choice: If the local NameNode is healthy, and ZKFC found no other nodes currently holding znode lock, it will acquire the lock for himself. If successful, it has won the selection, and is responsible for running the failover process so that its local NameNode is active. Failover and manual failover process described above is similar to the transfer, if the first active NameNode before the necessary protection, then the local NameNode converted to active status.
