Hadoop Sequence File 文件的读取和写入

版权声明：本文为博主原创文章，未经博主允许不得转载。 https://blog.csdn.net/leishenop/article/details/71439174

写入代码

下面是写入了100条(key,value)的信息，其中以LongWriable为key，以Text作为value.

        Configuration config = new Configuration();
        FileSystem fs  = FileSystem.get(conf);
        int i = 0;
        Path path = new Path("/home/lake/hello.xml");
        SequenceFile.Writer writer = null;
        SequenceFile.Writer.Option optPath = SequenceFile.Writer.file(path);
        //定义key
        SequenceFile.Writer.Option optKey = SequenceFile.Writer.keyClass(LongWritable.class);
        //定义value
        SequenceFile.Writer.Option optVal = SequenceFile.Writer.valueClass(Text.class);

        writer = SequenceFile.createWriter(conf, optPath, optKey, optVal);
        //写入的数据可以根据你的情况来定，我这只是测试
        String value = "hello world";

        while(i < 100){
            writer.append(new LongWritable(i),new Text(value));
            i ++;
        }
        writer.close();

上面程序运行完成之后，就可以在指定的路径上看到产生的文件。

读取的代码

                Configuration config = new Configuration();
                FileSystem fs  = FileSystem.get(conf);
                Path path = new Path("/home/lake/hello.xml");
                SequenceFile.Reader reader = new SequenceFile.Reader(fs.getConf(), SequenceFile.Reader.file(path));
                List<Object> sampleValues = new ArrayList<Object>();
                Writable key = (Writable) ReflectionUtils.newInstance(reader.getKeyClass(), fs.getConf());
                Writable value = (Writable) ReflectionUtils.newInstance(reader.getValueClass(), fs.getConf());
                int count = 0;
                String keyName = "Key";
                String valueName = "Value";
                //change data to json format
                while (reader.next(key, value) && count < 12) {
sampleValues.add("{\"" + keyName + "\": \"" + key + "\", \"" + valueName + "\": \"" + value + "\"}");
                    count++;
                }