上篇文章我们分析了open函数,这个函数会获取要打开文件的块信息,接下来我们开始分析读文件部分的代码。
我们先来看一个示例,代码如下:
package com.hadoop.senior.hdfs;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;
public class HDFSApp {
public static FileSystem getFileSystem() throws IOException{
//core-site.xml hdfs-site.xml log4j.properties
Configuration configuration=new Configuration();
//FileSystem
FileSystem filesystem= FileSystem.get(configuration);
return filesystem;
}
//读取hdfs文件系统上的文件,在窗口打印出来
public static void readFile(String filename) throws IOException{
FileSystem filesystem= getFileSystem();
//read path
Path path = new Path(filename);
//dfInputStream
FSDataInputStream inStream= filesystem.open(path);
try{
IOUtils.copyBytes(inStream, System.out, 4, false);
}catch(Exception e){
e.printStackTrace();
}finally{
IOUtils.closeStream(inStream);
}
}
//将本地文件上传到hdfs文件系统上
public static void writeFile(String filename) throws IOException{
FileSystem filesystem=getFileSystem();
Path path= new Path(filename);
FSDataOutputStream outStream=filesystem.create(path);
FileInputStream inStream=new FileInputStream(new File("/opt/modules/hadoop-2.5.0-cdh5.3.6/input.txt"));
try{
IOUtils.copyBytes(inStream, outStream, 1024, false);
}catch(Exception e){
e.printStackTrace();
}finally{
IOUtils.closeStream(inStream);
IOUtils.closeStream(outStream);
}
}
public static void main(String[] args) throws IOException {
//read local file not use core-site.xml hdfs-site.xml
//String filename="/opt/modules/workspace/senior/pom.xml";
//read hdfs input.txt
String filename ="/usr/css/mapreduce/wordcount/input/input.txt";
String filename2="/usr/css/mapreduce/wordcount/input/input02.txt";
readFile(filename);
writeFile(filename2);
}
}
上面这段代码有读和写两个功能,我们先来分析读,首先会先调用FileSystem类的open函数打开文件,这个open函数的分析见hadoop2.6.0源码剖析-客户端(第二部分--读(open)HDFS文件),然后执行代码
IOUtils.copyBytes(inStream, System.out, 4, false);
这行代码用来将文件内容从inStream中拷贝到System.out中,我们进入这个函数,代码如下:
/**
* Copies from one stream to another.
*
* @param in InputStrem to read from
* @param out OutputStream to write to
* @param buffSize the size of the buffer
* @param close whether or not close the InputStream and
* OutputStream at the end. The streams are closed in the finally clause.
*/
public static void copyBytes(InputStream in, OutputStream out, int buffSize, boolean close)
throws IOException {
try {
copyBytes(in, out, buffSize);
if(close) {
out.close();
out = null;
in.close();
in = null;
}
} finally {
if(close) {
closeStream(out);
closeStream(in);
}
}
}
我们看看copyBytes函数,代码如下:
/**
* Copies from one stream to another.
*
* @param in InputStrem to read from
* @param out OutputStream to write to
* @param buffSize the size of the buffer
*/
public static void copyBytes(InputStream in, OutputStream out, int buffSize)
throws IOException {
PrintStream ps = out instanceof PrintStream ? (PrintStream)out : null;
byte buf[] = new byte[buffSize];
int bytesRead = in.read(buf);
while (bytesRead >= 0) {
out.write(buf, 0, bytesRead);
if ((ps != null) && ps.checkError()) {
throw new IOException("Unable to write to output stream.");
}
bytesRead = in.read(buf);
}
}
DFSInputStream 继承 FSInputStream
FSInputStream 继承 InputStream
DFSOutputStream 继承 FSOutputSummer
FSOutputSummer 继承 OutputStream
这里由于in实际类型是DFSInpuStream,所以in.read(buf)会调用DFSInpuStream中的read函数。该函数代码如下:
/**
* Read the entire buffer.
*/
@Override
public synchronized int read(final byte buf[], int off, int len) throws IOException {
ReaderStrategy byteArrayReader = new ByteArrayStrategy(buf);
return readWithStrategy(byteArrayReader, off, len);
}
其中ByteArrayStrategy类代码如下:
/**
* Used to read bytes into a byte[]
*/
private static class ByteArrayStrategy implements ReaderStrategy {
final byte[] buf;
public ByteArrayStrategy(byte[] buf) {
this.buf = buf;
}
@Override
public int doRead(BlockReader blockReader, int off, int len,
ReadStatistics readStatistics) throws ChecksumException, IOException {
int nRead = blockReader.read(buf, off, len);
updateReadStatistics(readStatistics, nRead, blockReader);
return nRead;
}
}
在read函数中会调用readWithStrategy函数,该函数代码如下:
private int readWithStrategy(ReaderStrategy strategy, int off, int len) throws IOException {
//用来判断客户端连接是否断开了,如果断开了那么就抛出异常
dfsClient.checkOpen();
if (closed) {
throw new IOException("Stream closed");
}
Map<ExtendedBlock,Set<DatanodeInfo>> corruptedBlockMap
= new HashMap<ExtendedBlock, Set<DatanodeInfo>>();
failures = 0;
//如果当前读取的文件位置小于文件长度
if (pos < getFileLength()) {
//重试次数
int retries = 2;
while (retries > 0) {
try {
// currentNode can be left as null if previous read had a checksum
// error on the same block. See HDFS-3067
//如果当前读取的文件位置超过了块最大位置或者当前node为null,那么需要更新当前的node,
//也就是说此时要获取的数据不在当前的块上,而是另外一个块,所以需要找到需要的那个node
if (pos > blockEnd || currentNode == null) {
currentNode = blockSeekTo(pos);
}
int realLen = (int) Math.min(len, (blockEnd - pos + 1L));
if (locatedBlocks.isLastBlockComplete()) {
realLen = (int) Math.min(realLen, locatedBlocks.getFileLength());
}
int result = readBuffer(strategy, off, realLen, corruptedBlockMap);
if (result >= 0) {
pos += result;
} else {
// got a EOS from reader though we expect more data on it.
throw new IOException("Unexpected EOS from the reader");
}
if (dfsClient.stats != null) {
dfsClient.stats.incrementBytesRead(result);
}
return result;
} catch (ChecksumException ce) {
throw ce;
} catch (IOException e) {
if (retries == 1) {
DFSClient.LOG.warn("DFS Read", e);
}
blockEnd = -1;
if (currentNode != null) { addToDeadNodes(currentNode); }
if (--retries == 0) {
throw e;
}
} finally {
// Check if need to report block replicas corruption either read
// was successful or ChecksumException occured.
reportCheckSumFailure(corruptedBlockMap,
currentLocatedBlock.getLocations().length);
}
}
}
return -1;
}
当如果当前读取的文件位置超过了块最大位置或者当前node为null,那么需要更新当前的node,此时会调用函数blockSeekTo,我们进入到该函数中,代码如下:
/**
* Open a DataInputStream to a DataNode so that it can be read from.
* We get block ID and the IDs of the destinations at startup, from the namenode.
*/
private synchronized DatanodeInfo blockSeekTo(long target) throws IOException {
if (target >= getFileLength()) {
throw new IOException("Attempted to read past end of file");
}
// Will be getting a new BlockReader.
// 由于要获得一个新的BlockReader,所以这里需要将该变量置空
if (blockReader != null) {
blockReader.close();
blockReader = null;
}
//
// Connect to best DataNode for desired Block, with potential offset
//
DatanodeInfo chosenNode = null;
int refetchToken = 1; // only need to get a new access token once
int refetchEncryptionKey = 1; // only need to get a new encryption key once
boolean connectFailedOnce = false;
while (true) {
//
// Compute desired block
//
LocatedBlock targetBlock = getBlockAt(target, true);
assert (target==pos) : "Wrong postion " + pos + " expect " + target;
long offsetIntoBlock = target - targetBlock.getStartOffset();
//获取一个Datanode用来读取该数据块,
DNAddrPair retval = chooseDataNode(targetBlock, null);
chosenNode = retval.info;
InetSocketAddress targetAddr = retval.addr;
StorageType storageType = retval.storageType;
try {
ExtendedBlock blk = targetBlock.getBlock();
Token<BlockTokenIdentifier> accessToken = targetBlock.getBlockToken();
blockReader = new BlockReaderFactory(dfsClient.getConf()).
setInetSocketAddress(targetAddr).
setRemotePeerFactory(dfsClient).
setDatanodeInfo(chosenNode).
setStorageType(storageType).
setFileName(src).
setBlock(blk).
setBlockToken(accessToken).
setStartOffset(offsetIntoBlock).
setVerifyChecksum(verifyChecksum).
setClientName(dfsClient.clientName).
setLength(blk.getNumBytes() - offsetIntoBlock).
setCachingStrategy(cachingStrategy).
setAllowShortCircuitLocalReads(!shortCircuitForbidden()).
setClientCacheContext(dfsClient.getClientContext()).
setUserGroupInformation(dfsClient.ugi).
setConfiguration(dfsClient.getConfiguration()).
build();//构造从指定Datanode上读取数据块的BlockReader对象,这里使用了
//BlockReaderFactory工厂类
if(connectFailedOnce) {
DFSClient.LOG.info("Successfully connected to " + targetAddr +
" for " + blk);
}
return chosenNode;
} catch (IOException ex) {
if (ex instanceof InvalidEncryptionKeyException && refetchEncryptionKey > 0) {
DFSClient.LOG.info("Will fetch a new encryption key and retry, "
+ "encryption key was invalid when connecting to " + targetAddr
+ " : " + ex);
// The encryption key used is invalid.
refetchEncryptionKey--;
dfsClient.clearDataEncryptionKey();
} else if (refetchToken > 0 && tokenRefetchNeeded(ex, targetAddr)) {
refetchToken--;
fetchBlockAt(target);
} else {
connectFailedOnce = true;
DFSClient.LOG.warn("Failed to connect to " + targetAddr + " for block"
+ ", add to deadNodes and continue. " + ex, ex);
// Put chosen node into dead list, continue
addToDeadNodes(chosenNode);
}
}
}
}
函数blockSeekTo中调用getBlockAt用来获取指定位置所在的块,getBlockAt函数代码如下:
/**
* Get block at the specified position.
* Fetch it from the namenode if not cached.
*
* @param offset block corresponding to this offset in file is returned
* @param updatePosition whether to update current position
* @return located block
* @throws IOException
*/
private synchronized LocatedBlock getBlockAt(long offset,
boolean updatePosition) throws IOException {
assert (locatedBlocks != null) : "locatedBlocks is null";
final LocatedBlock blk;
//check offset
//数据偏移量不能小于0或者不能大于等于当前文件的大小,否则抛出异常
if (offset < 0 || offset >= getFileLength()) {
throw new IOException("offset < 0 || offset >= getFileLength(), offset="
+ offset
+ ", updatePosition=" + updatePosition
+ ", locatedBlocks=" + locatedBlocks);
}//如果偏移量大于等于当前文件大小
else if (offset >= locatedBlocks.getFileLength()) {
// offset to the portion of the last block,
// which is not known to the name-node yet;
// getting the last block
//获得最新的块数据
blk = locatedBlocks.getLastLocatedBlock();
}
else {
//在当前偏移量大于等于0且小于当前文件大小时,开始从缓存中读取响应块数据
// search cached blocks first
int targetBlockIdx = locatedBlocks.findBlock(offset);
if (targetBlockIdx < 0) { // block is not cached
//此时说明要找的偏移量数据不在指定的块中
targetBlockIdx = LocatedBlocks.getInsertIndex(targetBlockIdx);
// fetch more blocks
//从远程服务器上去获取指定偏移量的块数据
final LocatedBlocks newBlocks = dfsClient.getLocatedBlocks(src, offset);
assert (newBlocks != null) : "Could not find target position " + offset;
//将新获取到的块数据替换对应的老数据,对应规则下面会讲到
locatedBlocks.insertRange(targetBlockIdx, newBlocks.getLocatedBlocks());
}
//返回相应的块
blk = locatedBlocks.get(targetBlockIdx);
}
// update current position
if (updatePosition) {
pos = offset;
blockEnd = blk.getStartOffset() + blk.getBlockSize() - 1;
currentLocatedBlock = blk;
}
return blk;
}
我们进入到findBlock函数中,该函数用来获取指定偏移量所在块对应的索引号,在缓存中查找块索引值,代码如下:
/**
* Find block containing specified offset.
*
* @return block if found, or null otherwise.
*/
public int findBlock(long offset) {
// create fake block of size 0 as a key
LocatedBlock key = new LocatedBlock(
new ExtendedBlock(), new DatanodeInfo[0], 0L, false);
key.setStartOffset(offset);
key.getBlock().setNumBytes(1);
Comparator<LocatedBlock> comp =
new Comparator<LocatedBlock>() {
// Returns 0 iff a is inside b or b is inside a
@Override
public int compare(LocatedBlock a, LocatedBlock b) {
long aBeg = a.getStartOffset();
long bBeg = b.getStartOffset();
long aEnd = aBeg + a.getBlockSize();
long bEnd = bBeg + b.getBlockSize();
if(aBeg <= bBeg && bEnd <= aEnd
|| bBeg <= aBeg && aEnd <= bEnd)
return 0; // one of the blocks is inside the other
if(aBeg < bBeg)
return -1; // a's left bound is to the left of the b's
return 1;
}
};
return Collections.binarySearch(blocks, key, comp);
}
findBlock函数:
返回0表示一个块在另外一个块的里面,
返回-1表示要找到偏移量不在缓存中的块中
返回1表示其他情况
我们现在回到getBlockAt函数中,继续往下,我们进入到函数insertRange中,代码如下:
public void insertRange(int blockIdx, List<LocatedBlock> newBlocks) {
int oldIdx = blockIdx;
int insStart = 0, insEnd = 0;
for(int newIdx = 0; newIdx < newBlocks.size() && oldIdx < blocks.size();
newIdx++) {
long newOff = newBlocks.get(newIdx).getStartOffset();
long oldOff = blocks.get(oldIdx).getStartOffset();
if(newOff < oldOff) {
insEnd++;
} else if(newOff == oldOff) {
// replace old cached block by the new one
blocks.set(oldIdx, newBlocks.get(newIdx));
if(insStart < insEnd) { // insert new blocks
blocks.addAll(oldIdx, newBlocks.subList(insStart, insEnd));
oldIdx += insEnd - insStart;
}
insStart = insEnd = newIdx+1;
oldIdx++;
} else { // newOff > oldOff
assert false : "List of LocatedBlock must be sorted by startOffset";
}
}
insEnd = newBlocks.size();
if(insStart < insEnd) { // insert new blocks
blocks.addAll(oldIdx, newBlocks.subList(insStart, insEnd));
}
}
这个函数会将新的块替换老的块。我们回到getBlockAt函数中,获取到块后,会将相应的参数进行更新并返回块信息,当前的块读取偏移量,当前的块,当前块的最大偏移量。
至此整个读的过程就结束了!文章很多函数都没有深入讲解,后面会做相应的补充。