[Big Data Development Technology] Experiment 03-Hadoop Reading Files

Hadoop reads files

1. Experimental goals

1. Proficient in hadoop operating instructions and HDFS command line interface
2. Master HDFS principles
3. Master the API usage of HDFS
4. Master the method of reading file contents on HDFS through the URL class
5. Master how FileSystem reads file contents on HDFS

2. Experimental requirements

1. Provide screenshots of the successful results of each experimental operation step.
2. Provide a comprehensive summary of this experimental work.
3. After completing the experiment content, the experiment report file is renamed: Student ID Name Experiment 3.

3. Experimental content

1. Use the FileSystem class to read the file on HDFS, print the content of the file to the standard output stream, and test it locally and on the cluster respectively. The detailed implementation process, complete code and screenshots of the implementation effect are given. The final renderings refer to Figure 1 and figure 2. (It is required to create a new file locally. The file name or file content should reflect your name. The file content can be prepared in both Chinese and English. Use the shell command to upload it to HDFS.) Figure 1 Local test effect of FileSystem reading the file Figure 2
   
   FileSystem
   
   reading Fetch file cluster test renderings

2. Through the URL class, the file content on HDFS is read, and the main implementation process, complete code and implementation effect screenshots are given. For the final renderings, please refer to Figure 3 and Figure 4. (It is required to create a new file locally. The file name or file content should reflect your name. The file content should be prepared by yourself. Both Chinese and English can be used. Use the shell command to upload it to HDFS.) Figure 3 Local test effect of URL reading file Figure 4
   
   URL
   
   reading File cluster test renderings

3. Use the FileSystem class to read multiple files on HDFS, print the contents of the files to the standard output stream, and provide screenshots of the main implementation process, complete code, and implementation effects. See Figure 5 for a screenshot of the implementation effect (Figure 5 is a screenshot of the test effect of reading the contents of the two files cs.txt and cslg.txt). It is required to create two new files locally. Two of the files are named your student ID and name. The file contents are personal profile and hometown information. The file contents can be both Chinese and English. Use the shell command to upload to HDFS.
   
   Figure 5 FileSystem method to read the contents of multiple files

4. Through the URL class, the contents of multiple files on HDFS are read, and the main implementation process, complete code and implementation effect screenshots are given. The final renderings are shown in Figure 6. Using the two text files in the previous experiment, you can also create two new text files. The file names are the same as the previous experiment and the file contents are customized.
   
   Figure 6 Reading multiple file contents using URL

4. Experimental steps

Experiment 1

Experimental code

package com.wjw.hadoop;
import java.io.IOException;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;

public class FileSystemCat01 {
    
    

    public static void main(String[] args) {
    
    
        // TODO Auto-generated method stub
        String uri = "hdfs://master:9000/wjw01.txt";
        Configuration conf = new Configuration();
        conf.set("fs.defaultFS", "hdfs://master:9000");
        FileSystem fs = null;
        FSDataInputStream in =null;
        try{
    
    
            fs = FileSystem.get(conf);
            in = fs.open(new Path(uri));
            IOUtils.copyBytes(in, System.out, 4096, false);
            
        }catch(IOException e){
    
    
            e.printStackTrace();
        }finally{
    
    
            if(in != null){
    
    
                try{
    
    
                    fs.close();
                }catch(IOException e){
    
    
                    e.printStackTrace();
                }
            }
        }
        
    }

}

Experiment screenshot

Experiment 2

Experimental code

package com.wjw.hadoop;

import java.io.IOException;
import java.io.InputStream;
import java.net.URL;

import org.apache.hadoop.fs.FsUrlStreamHandlerFactory;
import org.apache.hadoop.io.IOUtils;

public class FileCat01 {
    
    
	
    static{
    
    
    	URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory());
     }
	public static void main(String[] args) {
    
    
		// TODO Auto-generated method stub
		String arg = "hdfs://master:9000/wjw02.txt";
      InputStream in = null;
      try{
    
    
          in = new URL(arg).openStream();
          IOUtils.copyBytes(in, System.out, 2048, false);
          }catch(IOException e){
    
    
        	  e.printStackTrace();
          }finally{
    
    
        	  IOUtils.closeStream(in);
          }
    }
}

Experiment screenshot

Experiment 3
Experiment Code

package com.wjw.hadoop;
import java.io.IOException;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;

public class FileSystemCat02 {
    
    

    public static void main(String[] args) {
    
    
        // TODO Auto-generated method stub
		args = new String[2];
		args[0] = "hdfs://master:9000/wjw03.txt";
		args[1] = "hdfs://master:9000/wjw04.txt";
        Configuration conf = new Configuration();
        conf.set("fs.defaultFS", "hdfs://master:9000");
		for(int i=0; i < args.length; i++){
    
    
        FileSystem fs = null;
        FSDataInputStream in =null;
        try{
    
    
            fs = FileSystem.get(conf);
            in = fs.open(new Path(args[i]));
            IOUtils.copyBytes(in, System.out, 4096, false);
            
        }catch(IOException e){
    
    
            e.printStackTrace();
        }finally{
    
    
            if(in != null){
    
    
                try{
    
    
                    fs.close();
                }catch(IOException e){
    
    
                    e.printStackTrace();
                }
            }
        }
		}
    }

}

Experiment screenshot

Experiment 4
Experiment Code

package com.wjw.hadoop;

import java.io.IOException;
import java.io.InputStream;
import java.net.URL;

import org.apache.hadoop.fs.FsUrlStreamHandlerFactory;
import org.apache.hadoop.io.IOUtils;

public class FileCat02 {
    
    
	
    static{
    
    
    	URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory());
     }
	public static void main(String[] args) {
    
    
		// TODO Auto-generated method stub
		args = new String[2];
		args[0] = "hdfs://master:9000/wjw03.txt";
		args[1] = "hdfs://master:9000/wjw04.txt";
		for(int i=0; i < args.length; i++){
    
    
          InputStream in = null;
          try{
    
    
          in = new URL(args[i]).openStream();
          IOUtils.copyBytes(in, System.out, 2048, false);
          }catch(IOException e){
    
    
        	  e.printStackTrace();
          }finally{
    
    
        	  IOUtils.closeStream(in);
          }
		}
	}

}

Experiment screenshot

Attachment: series of articles

experiment	Article directory	direct link
Experiment 01	Hadoop installation and deployment	https://want595.blog.csdn.net/article/details/132767284
Experiment 02	HDFS common shell commands	https://want595.blog.csdn.net/article/details/132863345
Experiment 03	Hadoop reads files	https://want595.blog.csdn.net/article/details/132912077
Experiment 04	HDFS file creation and writing	https://want595.blog.csdn.net/article/details/133168180
Experiment 05	Create, delete and query HDFS directories and files	https://want595.blog.csdn.net/article/details/133168734