Share Hadoop Big Data learning route high-availability configuration stage , what is Hadoop mechanism of HA
Ha mechanism that is Hadoop high availability ( 7 * 24 hours without service interruption)
HA mechanism is formally introduced hadoop2.0 from the beginning, the previous version is no mechanism for HA
hadoop-ha strictly speaking be divided into individual components HA mechanism - the HDFS of HA , YARN of HA
Detailed mechanisms of HDFS HA
HDFS 's HA mainly through the double namenode coordination to achieve
Double namenode coordination of points:
A , metadata management needs to change:
Memory, keep a journal metadata
Edits log can only have one, and only Active state namenode node can do write operations
Two namenode can read edits
Share of edits in a shared storage management ( qjournal and NFS two mainstream achieve)
B , need a state management function module
Implements a zkfailover , every resident in namenode node is located
Each zkfailover responsible for monitoring where their namenode node, using the zk the state logo
When the state switching required by zkfailover responsible for switching
When the switch is required to prevent brain split split-brain phenomenon occurs
Namenode operating principle of
There is a server on both the NameNode , one of which Namenode in an active state, in a standby state, two servers to share data, each of the two servers there a piece of metadata, edit data but only one, only two servers namenode server is in active state can edit write, another server can only edit read, and manage sharing edit into a shared storage. Shared storage is implemented by the file management system qjournal and NFS.
The two servers active standby state how to manage, you will need a management module: ZKFC (zookeeper failover controller) to manage. Each zkfc responsible for monitoring where their namenode node, using the zk the state identity. When the state switching required by zkfailover responsible for switching
When the switch is required to prevent brain split split brain occurs phenomenon .
What is the split-brain phenomenon
Split-brain phenomenon is both namenode are in A ctive state of conflict, this is the split brain. Hadoop high-availability configuration to be solved split brain state.
How to split brain state generator
When an active state namenode server in suspended animation, then another name the Node server zkfc received information belonging to his namenode state is changed to active state, the first in a state of suspended animation namdenode they wake up, they will produce split brain.
How to solve the split brain
A second namenode of zkfc case will lamb, in the first state of suspended animation namenode kill using ssh kill -9 namenode, directly kill the first server namenode be up a knife, the knife up if unsuccessful, zkfc into the first server, the user directly call custom scripts /home/Hadoop/kill/poweroff.sh killing - out false - dead namenode.