Their duties
To give up on chestnuts:
the NameNode we say here is referred to as the president of NN
Datanode staff referred DN
Secondary vice president referred to the NameNode SNN
Why have the vice president do? The boss hands live too much, a person needs help, he can turn the confidential documents to the Secretary what, obviously not, so it appeared the vice president.
NN master a number of metadata (metadata that describes data, such as database fields is to describe the data)
have data yet, it is necessary to ensure the safety of metadata -------- data stored in the memory to disk ------ this is achieved persistence
RAM
Here simply under the memory
hard disk memory is large, inexpensive, but slow
small memory memory, expensive, but fast
safe question
In this way it is:
When you cluster because of our special reasons or the power generated asked me to solve the problem, reboot, reads the metadata to disk, restore to the former state
reasons can not be persistent NN
1. do: demand for small, small memory footprint, does not affect the calculation of the efficiency of
case 2. can not do: NN itself has a lot of work, it may be persistent during downtime.
Note: SNN never replace the position of NN, NN, he was just a hot standby
to a straightforward graphic interpretation
to explain some
edits to store the operating system running process information generated in China.
fsimage contains information about all the directories and files of the entire HDFS file system. For a file, the data block including description information, modification time, access time and the like; For a directory includes a modification time, access control information (directory belongs to the user, where the group) and the like.
Mainly edits log information generated by the operation. If edits have not done anything, it will continue to produce the information.
The first step: hdfs update records are written to a new file --edits.new.
Step two: fsimage and editlog sent to the secondary namenode via http protocol.
The third step: fsimage and editlog merge, creating a new file --fsimage.ckpt. The reason for this step in the secondary namenode in, because it is more time-consuming, if done in namenode or cause the entire system Caton.
The fourth step: to transmit the generated fsimage.ckpt namenode via http protocol.
Step five: Rename fsimage.ckpt as fsimage, edits.new to edits.
Repeating such a 1,, 2,3,4,5 step.
So if namenode down, in fact, secondary namenode also holds that this is a recent fsimage, can recover some losses.
Persistent trigger condition
*超过3600s或者edits的大小超过64m*
Then there was another case, if we put fsimage edits and submit to the SNN, NN's etids consistent storage, full of how to do?
It does not matter, my brother give you a map
solutions
1. isolated phenomenon
Start a separate edits, which exist in two edits
2. The normal
need for cluster adjustment, adjust the size of edits
summary : persistence is to write metadata NN into disk storage, after the restart when hung back NN disk reads the corresponding metadata, cluster ----- recovery state (off loss of memory)
a power outage
before persistent ----- start again, read the system log
----- read the data on the disk after persistent, restore the state
repeat off
communication mechanism of NN and DN ---- heartbeat mechanism (every 3s, DN send back once to NN beat 1 minute not think hang on DN)
safe mode
1. restore the system status
2. check the information DN
3 DN fix a problem
1) power failure during transmission - loss of data
if the data is particularly important that only in advance and pre-judgment, make the appropriate adjustments
2) off after the transfer is complete
when I resumed cluster, NN is not going to read the metadata, the state of the corresponding recovery
3) after recovery in DN, If any of the new In some cases, determine whether the new file upload
for 3) in this case we look at this explain
a. When uploading block1 DN3 waste, the current task will not be uploaded
b. When DN3 After recovery, corresponding to a new node.
When c. publish new task will be written on DN3 recovery
d. When the data prior to hang on computing DN3, would go on to find 1,2.
We welcome comments and guidance