(Ii) the HDFS data stream
As a file system, read and write files is the most basic needs, this part of our clients is to understand how to interact with HDFS, that is, the client and HDFS, and HDFS composed of two types of nodes (namenode and datanode) data flow between what.
1, to parse the document reading process
HDFS client read from a file, the internal reading process is actually more complicated, can be represented by the following basic flow of FIG read the file.
For the client, the first thing is to call the FileSystem object's open () method to open the file you wish to read , then DFS returns a file input stream FSDataInputStream, the input stream to the client calls the read () method reads the data , upon completion of reading, the input stream to this call close () method to close, these three processes of 1,3,6 corresponding steps of FIG.
The above three steps from the perspective of the client to analyze, in fact, to achieve the file read inside HDFS also need to support more complex mechanisms, and these processes are transparent to the client , so the client feel , it seems as if the customer is reading a continuous stream.
Specifically, from the perspective of HDFS, Open client calls the object FileSystem () method, the object is actually FileSystem distributed file system is one example of Distributed File, Distributed File be via remote procedure calls (RPC) calls NameNode, to file start block location is obtained (step 2, namenode there datanode return address of the data block copy). Of course, since multiple copies of a HDFS saved data block (default is 3), the request is satisfied datanode than one address, then will be sorted according to their distance from the client, preference datanode near distance, if the client end in itself is a datanode, the client can read the data from the local (for example: mapReduce on the use of the advantages of localization data here).
After completion of the open method, DistributedFileSystem class returns a FSDataInputStream file input stream object to the client. This class in turn encapsulated as DFSInputStream object that manages the namenode datanode and I / O.
This DFSInputStream files storing the start address datanode several blocks, and therefore, the client calls the read of the input stream () method can be used to know to which datanode (latest network topology distance) to read data, so called repeatedly the method can read the data from datanode to the client (step 4). When reaching the end of a block, and closes the connection datanode the search for the next best block datanode, repeat the process.
Of course, we said above DFSInputStream only a few blocks of the file is stored starting in the reading process, will also ask namenode again as needed to get the next batch of data datanode address block. Once the client has completed the reading, it calls the close method Close the stream.
If the reading process, datanode experience a failure, it is clear that the input stream just from another recently saved datanode can read a copy of the data block, bearing in mind that failure datanode, after avoid reading data from there .
to sum up:
These are the HDFS document reading process, from the analysis of this process we can see: the advantage that the client may connect directly to datanode read data, so that since the data dispersed in different datanode, can be simultaneously a large number of concurrent client service. Namenode as the management node, only the response request data block location, the client can inform the best datanode where each data block (datanode position information stored in memory, can obtain a very efficient). This makes namenode without the need for specific data transmission tasks, otherwise namenode in the number of clients of the situation will become a bottleneck.
2, analysis of file write process
Next, we analyze the process of writing a file, a key consideration is how the case to create a new file, how to write data to the file and finally close the file.
By the same token, from the client's perspective, this process is relatively simple, first by calling on DistributedFileSystem create objects () method to create a new file, and then returns a FSDataOutputStream file output stream object, which the client can call after writing the write data output stream () method, the writing is completed, calling close () method Close (1,3,6 steps in the chart) output stream.
然而，具体的，从HDFS的角度来看，这个写数据的过程就相当复杂了。客户端在调用create方法新建文件时，DistributedFileSystem会对namenode创建一个RPC调用，在文件系统的命名空间中新建一个文件，此时还没有相应的数据块（步骤2）。namedata接收到这个RPC调用后，会进行一系列的检查，确保这个文件不存在，并且这个客户端有新建文件的权限，然后再通过检查后就会为这个新文件在命名空间中加入一条记录（如果未通过检查则会返回异常），最后给客户端返回一个FSDataOutputStream对象。
类似于文件读的过程，这个FSDataOutputStream对象转而封装成为一个DFSOutputStream对象，用于处理datanode和namenode之间的I/O。
接下来，客户端就可以调用输出流的write()方法进行数据写入，而在写入时，DFSOutputStream将数据分为一个一个的数据包，先写入内部队列，称为“数据队列”。然后有一个单独的DataStreamer来处理数据队列，它的职责是挑选出适合存储数据副本的一组datanode，并要求namenode分配新的数据块。假设副本数为3，那么选出来的datanode就是3个，这3个dadanode会构成一个数据管线。DataStreamer将数据包流式传输到管线中的第一个datanode，第一个存储并发到第二个，第二个存储并发到第三个（步骤4）。
然后DFSOutputStream对象内部还有一个数据包队列用于接收datanode的确认回执，称为“确认队列”，收到所有datanode的确认消息后，该数据包才会从队列中删除。
在客户端完成数据的写入后，对数据流调用close()方法（步骤6），该操作将剩余的所有数据包写入数据管线，并联系namenode告知文件写入完成之前，等待确认（步骤7）。
3、一致模型
HDFS的一致模型描述了文件读、写的数据可见性。
基于以上对文件读写过程的分析，我们知道新建一个文件之后，它可以在命名空间中立即可见，但是即使数据流已经刷新并存储，写入文件的内容并不保证能立即可见。当写入的数据超过一个数据块后，第一个数据块对新的reader就是可见的，也就是说：当前正在写的块对其他reader不可见。
HDFS提供了一种将所有缓存刷新到datanode中的方法，即对FSDataOutputStream调用hflush()方法，当hflush方法调用成功后，到目前为止写入的数据都到达了datanode的写入管道并且对所有reader可见。
但是，hflush()并不保证数据已经都写到磁盘上，为确保数据都写入磁盘，可以使用hsync()操作代替。
在HDFS中，close方法实际上隐含了执行hflush()方法。
4、通过distcp并行复制
当我们想从Hadoop文件系统中复制大量数据或者将大量数据复制到HDFS中时，可以采用Hadoop自带的一个程序distcp，它用来并行复制。
distcp的一个用法是代替hadoop fs -cp，也可以用来在两个HDFS集群之间传输数据。
$ hadoop distcp file1 file2
$ hadoop distcp dir1 dir2
$ hadoop distco -update -delete -p hdfs://namenode1/foo hdfs://namenode2/foo总结
以上主要对HDFS文件系统的文件读写进行了详细的介绍，重点是掌握HDFS的文件读写流程，体会这种机制对整个分布式系统性能提升带来的好处。