01 Let's learn big data together
Lao Liu continued to share some advanced knowledge points of the HDFS module in Hadoop, and it can be regarded as a summary of the HDFS content reviewed today. He hopes to give some help to students who want to learn big data, and also hope to get criticism and criticism from big guys. Give pointers!
02 knowledge points
Point 10: Heartbeat mechanism of HDFS mechanism
According to this figure, let's talk about the working principle of the heartbeat mechanism. First, when the master starts, an ipc server will be opened there; then after the slave starts, it will register a connection with the master every 3 seconds The clock sends a heartbeat to the master, carrying status information; finally, the master will use the return value of this heartbeat to convey instructions to the slave node.
After talking about the workflow, what is the use of the heartbeat mechanism now ?
1. NameNode periodically receives heartbeat signals and block status reports from each DataNode in the cluster. Receiving the heartbeat signal means that the DataNode is working normally. The block status report contains a list of all data blocks on this DataNode.
2. After the DataNode starts, it registers with the NameNode, and then periodically reports all the block lists to the NameNode; sends a heartbeat to the NameNode every 3 seconds, and returns the command from the NameNode to the DataNode. If the NameNode does not receive a heartbeat from a DataNode for more than 10 minutes, the node is considered unavailable.
3. When the Hadoop cluster first starts up, it will enter the safe mode (99.9%), and the heartbeat mode will be used.
Lao Liu has seen HDFS courseware from several institutions, and most of them have taken security mode at once. In Lao Liu's view, the safe mode must at least know its concept!
What is safe mode?
The security mode is a protection mechanism of Hadoop to ensure the security of data blocks in the cluster. In this state, the file system only accepts data read requests, and does not accept changes such as deletion and modification.
When the Hadoop cluster starts to enter the safe mode, it will check the integrity of the data blocks. Assuming that the number of replicas we set is 5, there should be 5 replicas on the DataNode. But again, assuming there are only 3 at present, 3/5=0.6 is less than the minimum replica probability of 99.9%, then the system will automatically replicate the replicas to other DataNodes. If there are 8 in the system and we only set 5, then the extra 3 will be deleted. Under normal circumstances, the safe mode will automatically exit after running for a period of time.
Point 11: HDFS data read process
What is its basic process ?
Let's take a look at the diagram. In the HDFS reading process, the client calls the open method of the file system, and then the file system will remotely call the open method in the NameNode through RPC to get the block location information returned. After the block location information is returned, the NameNode will call the read method of FSDataInputStream, and it will contact its nearest DataNode (this recently refers to the network topology for sorting, and the closest to the client is ranked first. This is not very sure, you can go to Baidu to see Look). If the first DataNode cannot be connected, the client will automatically contact the next DataNode; if the check value of the block data is wrong, the client needs to report to the NameNode and automatically contact the next DataNode.
Fault tolerance of data reading
What should I do if the communication between Client and DataNode is interrupted during block reading?
The client establishes a connection with another DataNode storing this block, reads the data, and records the problematic DataNode, and will not read data from it.
What should I do if the client reads the block and finds that there is a problem with the block data?
When we store the data, it will contain the check checksum (CRC32-), and also read the checksum when reading, and calculate a value when reading, and compare whether the two values are equal. If they are not equal, which node block has a problem, read from another node, tell the NameNode which node block has the problem, and copy a copy of data from other nodes to this node.
What should I do if the data read by the client is incomplete?
If they are not equal, which node has a problem with the block block, another node will read it, tell the NameNode which node has the block block, and copy a copy of data from other nodes to this node.
Let's take a look at the diagram. In the HDFS reading process, the client calls the open method of the file system, and then the file system will remotely call the open method in the NameNode through RPC to get the block location information returned. After the block location information is returned, the NameNode will call the read method of FSDataInputStream, and it will contact its nearest DataNode (this recently refers to the network topology for sorting, and the closest to the client is ranked first. This is not very sure, you can go to Baidu to see Look). If the first DataNode cannot be connected, the client will automatically contact the next DataNode; if the check value of the block data is wrong, the client needs to report to the NameNode and automatically contact the next DataNode.
Data read fault tolerance
What should I do if the communication between Client and DataNode is interrupted during the block reading process?
The client establishes a connection with another DataNode storing this block, reads the data, and records the problematic DataNode, and will not read data from it.
What should I do if the client reads the block and finds that there is a problem with the block data?
When we store the data, we will include the check checksum (CRC32-), and also read the checksum when reading, and calculate a value when reading, and compare whether the two values are equal. If they are not equal, which node block has a problem, read from another node, tell the NameNode which node block has the problem, and copy a copy of data from other nodes to this node.
What should I do if the data read by the client is incomplete?
If they are not equal, which node has a problem with the block block, another node will read it, tell the NameNode which node has the block block, and copy a copy of data from other nodes to this node.
Point 12: HDFS data writing process
What is its basic process?
1. The client calls the create method of the distributed system, and the file system also calls the create method of the NameNode remotely via RPC. At this time, the NameNode will take the following actions: ①Check whether it is running normally ②Determine whether the file to be created exists ③Whether the client has the permission to create the file ④The status change of HDFS needs to be recorded in the edits log
2. After the check is passed, the NameNode responds that the client can upload.
3. The client starts uploading the first block according to the block size set by itself, the default is 0-128M, and the NameNode selects a specified number of DataNode nodes according to the rack-aware strategy according to the number of copies of the file uploaded by the client (default is 3) return.
4. The client requests the establishment of a transmission channel according to the returned DataNode. The client initiates a channel establishment request to the nearest DataNode node, and then this DataNode node sequentially sends a channel establishment request to the next node in the channel (the closest distance to the current DN), and each node sends a response, and the channel establishment is successful.
5. Each time the client reads 64K of data, it is encapsulated into a packet (it is a data packet and the basic unit of transmission), and the packet is sent to the next node of the channel. After the node in the channel receives the packet, it will place the disk Store and send the packet to the next node in the channel. After receiving the packet, each node sends an ack confirmation message to the client.
6. After the data transmission of a block is completed, the channel is closed, and the DataNode reports a message to the NameNode that a block has been received
7. The transmission of the first block is completed, and the transmission of the second block is started. Repeat the above steps in sequence until the transmission of the last block is completed. The NameNode responds to the client that the transmission is completed, and the client closes the output stream.
But what should I do if there is an abnormality in the writing process?
Steps 1 to 4 are the same as before, look directly above.
5. Each time the client reads 64K of data, it is encapsulated into a packet, and the successfully encapsulated packet is put into a queue. This queue is called dataQuene (data packet to be sent).
6. When sending, first send the packets in dataQuene in order, and then put them into ackquene (the sending queue) after sending. After each node receives the packet, it sends an ack confirmation message to the client.
If a packet has received ack confirmation messages returned by all DNs after being sent, the packet will be deleted in ackquene.
If a packet times out after receiving the ack confirmation message returned by the DN after being sent, the transmission is aborted, and the packet in the ackquene will roll back to dataQuene. Re-establish the channel and remove the bad DN node. After the establishment is complete, continue the transmission. As long as a DN node receives the data and the DN reports that the NN has received the block, NN considers that the current block has been successfully transmitted.
Point 13: Hadoop HA
The high availability of Hadoop has already been discussed in ZooKeeper. The link is given directly. You can check it out. I want to enter
the 14th point of the big data ZooKeeper knowledge point (2) of the big factory .
Point 14: Hadoop Federation
For a very large cluster with a large number of files, memory will become a bottleneck restricting the horizontal development of the system. Federation is created to break through this bottleneck.
Point 15: Why is HDFS not suitable for storing small files?
First give some answers found on the Internet:
First of all, the NameNode stores the metadata of the file system. Each file, directory, and block has about 150 bytes of metadata. Therefore, the limit on the number of files is also determined by the memory size of the NameNode. If there are too many small files, it will cause excessive pressure on the NameNode. Big.
Let me give my own summary:
HDFS was born to store large files. The size of a block of data is about 150 bytes, and storing a small file will take up 150 bytes of memory. If you store a large number of small files, you will quickly run out of memory, but the amount of data stored in the entire cluster is small, and the meaning of HDFS is lost.
How to solve the problem of storing a large number of small files?
Using the Sequence File solution, the core is to organize small files with the file name as the key and the file content as the value. A large number of small files can be programmed to convert these files into a Sequence File file, and then the files can be processed in a data stream. Use MapReduce for processing. The advantages of this: ① Sequence File can be divided, MapReduce can divide the file into blocks, and each block can be operated independently. ②Sequence File supports compression. In most cases, it is best to compress by block, because a block contains multiple records, and the similarity between records is used for compression, and the compression efficiency is higher.
03 Summary
Alright! The summary of the knowledge points of Hadoop HDFS is over. Lao Liu shared these knowledge points, one is to review the knowledge points of HDFS, the other is to help students who want to learn big data, and the third is to get criticism from big guys. And pointing.
Finally, if there is something, the official account: Lao Liu who is working hard, contact; if nothing is wrong, I will learn big data with Lao Liu.