Hadoop series (four) HDFS's three core Hadoop Java API

The main purpose of HDFS is designed to store huge amounts of data, which means that it can be stored on a very large number of files.

After HDFS split these files stored on different DataNodes, HDFS functionality provided by the Java API to operate on the inside HDFS file, data blocks in the storage position on DataNodes, is transparent to developers.

Use Java API can perform various operations on HDFS, such as a new file, delete the file, read the file content. The following describes HDFS common Java API and programming examples.

concept

• Configuration package configuration of the client or server
• FileSystem file system object, the object is a method to perform file operations
• FileStatus for meta data to the client display system files and directories
• FSDatalnputStream HDFS in the input stream, for reading the file Hadoop
• The output stream FSDataOutputStream HDFS, Hadoop file for writing
• Path represents a path for Hadoop file system files or directories

Specific operations

• Introduction maven dependencies

<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-client</artifactId>
    <version>3.1.1</version>
</dependency>

• 2 Initial Configuration

Configuration conf;
FileSystem fileSystem;

public HdfsAPI() {
    conf = new Configuration();
    conf.set("dfs.replication", "2");
    conf.set("dfs.blocksize", "128m");
    try {
        fileSystem = FileSystem.get(new URI("hdfs://${NameNode}:9000"), conf, "hadoop");
    } catch (Exception e) {
        e.printStackTrace();
    }

}

• 3 get files get

public void testGet() throws IllegalArgumentException, IOException {
    fileSystem.copyToLocalFile(new Path("/output/part.txt"), new Path("~/Downloads"));
    fileSystem.close();
}

• 4 Getting file information ls

public void testLs() throws IllegalArgumentException, IOException {
    RemoteIterator<LocatedFileStatus> listFiles = fileSystem.listFiles(new Path("/"), true);
    while (listFiles.hasNext()) {
        LocatedFileStatus status = listFiles.next();
        System.out.println("路径：" + status.getPath());
        System.out.println("块大小：" + status.getBlockSize());
        System.out.println("文件长度：" + status.getLen());
        System.out.println("副本数:" + status.getReplication());
        System.out.println("块的位置信息：" + Arrays.toString(status.getBlockLocations()) + "\n");
    }
    fileSystem.close();
}

• 5. Create the directory, multi-level directory mkdir

public void testMkdir() throws IllegalArgumentException, IOException {
    fileSystem.mkdirs(new Path("/output/test/testmk"));
    fileSystem.close();
}

• 6 delete files, directories rm

public void testDeldir() throws IllegalArgumentException, IOException {
    boolean delete = fileSystem.delete(new Path("/output/test/testmk"), true);
    if (delete) {
        System.out.println("文件已经删除");
    }
    fileSystem.close();
}

• 7 reads the contents of the file hdfs

public void testReadData() throws IOException {
    FSDataInputStream in = fileSystem.open(new Path("/test.txt"));//hdfs自带流打开文件
    BufferedReader br = new BufferedReader(new InputStreamReader(in, "utf-8"));//读入流并放在缓冲区
    String line = null;
    while ((line = br.readLine()) != null) {
        System.out.println(line);
    }

    in.close();
    br.close();
    fileSystem.close();
}

• Hdfs 8 reads the contents of the file specified offset

public void testRandomReadData() throws IOException {
    FSDataInputStream in = fileSystem.open(new Path("/test.txt"));
    in.seek(12);//定位到12位置开始读
    byte[] buf = new byte[16];//往后读取16个位
    in.read(buf);//ba流读到buf中
    System.out.println(new String(buf));
    in.close();
    fileSystem.close();
}

• 9 write the data in hdfs

public void testWriteData() throws IOException {
    FSDataOutputStream out = fileSystem.create(new Path("/yy.jpg"), false);
    FileInputStream in = new FileInputStream("~/Download/wechatpic_20190309221605.jpg");
    byte[] buf = new byte[1024];
    int read = 0;
    while ((read = in.read(buf)) != -1) {
        out.write(buf, 0, read);
    }
    out.close();
    fileSystem.close();
}