JAVA-based HDFS file operations

JAVA-based HDFS file operations

First, the experiment content

1, a programming class "MyFSDataInputStream", which inherits "org.apache.hadoop.fs.FSDataInputStream", the following requirements: a row reading method implemented in HDFS specified file "readLine ()", if the end of file read returns empty, otherwise the text file line.

2, view the Java help manuals or other materials with "java.net.URL" and "org.apache.hadoop.fs.FsURLStreamHandler
Factory's" complete programming output HDFS file specified text to the terminal.

Second, the experiment code

1、

package Second;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
public class MyFsDataInputStream extends FSDataInputStream{
	public MyFsDataInputStream(InputStream in) {
		super(in);
	}
	public static String readline(Configuration conf,String filename) throws IOException
	{
		Path filename1=new Path(filename);
		FileSystem fs=FileSystem.get(conf);
		FSDataInputStream in=fs.open(filename1);
		BufferedReader d=new BufferedReader(new InputStreamReader(in));
		String line=d.readLine();
		if (line!=null) {
			d.close();
			in.close();
			return line;
		}else
			return null;
	}
	public static void main(String[] args) throws IOException {
		Configuration conf=new Configuration();
		conf.set("fs.defaultFS", "hdfs://localhost:9000");
		conf.set("fs.hdfs.impl", "org.apache.hadoop.hdfs.DistributedFileSystem");
		FileSystem fs=FileSystem.get(conf);
		String filename="/user/hadoop/myLocalFile.txt";
		System.out.println("读取文件："+filename);
		String o=MyFsDataInputStream.readline(conf, filename);
		System.out.println(o+"\n"+"读取完成");
	}
}

2、

package Second;
import java.io.IOException;
import java.io.InputStream;
import java.net.MalformedURLException;
import java.net.URL;
import org.apache.hadoop.fs.FsUrlStreamHandlerFactory;
import org.apache.hadoop.io.IOUtils;
public class FSUrl {
	static {
		URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory());
	}
	public static void cat(String filename) throws MalformedURLException, IOException
	{
		InputStream in=new URL("hdfs","localhost",9000,filename).openStream();
		IOUtils.copyBytes(in, System.out,4096,false);
		IOUtils.closeStream(in);
	}
	public static void main(String[] args) throws MalformedURLException, IOException {
		String filename="/user/hadoop/myLocalFile.txt";
		System.out.println("读取文件"+filename);
		FSUrl.cat(filename+"\n读取完成");
	}
}

Third, the operating results

1, a programming class "MyFSDataInputStream", which inherits "org.apache.hadoop.fs.FSDataInputStream", the following requirements: a row reading method implemented in HDFS specified file "readLine ()", if the end of file read returns empty, otherwise the text file line.

(1), launch a browser to view Namenode, Datanode node information, and file information
note you need to start hadoop in the terminal

(2), content from a browser to view the document

(3), run the code results eclipse

2, view the Java help manuals or other materials with "java.net.URL" and "org.apache.hadoop.fs.FsURLStreamHandlerFactory" programming is complete HDFS output file specified text to the terminal.

Content (1), view files

(2), the results of running the code in eclipse