MapReduce【自定义OutputFormat】

目录

自定义OutPutFormat

步骤

案例

需求

输入数据

 LogMapper类

 LogReducer类

MyOutputFormat类

MyRecordWriter类

LogRunner类

运行结果

---

MapReduce默认的输出格式为TextOutputFormat，它的父类是FileOutputFormat，即按行来写，且内容写到一个文本文件中去,但是并不能满足我们实际开发中的所有需求，所以就需要我们自定义OutPutFormat。

自定义OutPutFormat

输出数据到MySQL、HBase或者Elasticsearch等存储框架中。

步骤

继承FileOutputFormat

继承RecordWriter类

重写write方法

案例

需求

过滤log日志，将包含sxau的网站输出到sxau.log，其他则输出到other.log

输入数据

www.baidu.com
www.google.com
www.sxau.com
www.jd.com
www.bing.com
www.sina.com
www.csdn.com
www.github.com

 LogMapper类

public class LogMapper extends Mapper<LongWritable, Text,Text, NullWritable> {
    @Override
    protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {

        //直接写出
        context.write(value,NullWritable.get());

    }
}

 LogReducer类

public class LogReducer extends Reducer<Text, NullWritable,Text,NullWritable> {

    @Override
    protected void reduce(Text key, Iterable<NullWritable> values, Context context) throws IOException, InterruptedException {

        //防止相同数据丢失
        for (NullWritable value : values) {
            context.write(key,NullWritable.get());
        }

    }
}

MyOutputFormat类

 返回我们自定义的RecordWriter类

public class MyOutputFormat extends FileOutputFormat<Text, NullWritable> {

    @Override
    public RecordWriter<Text, NullWritable> getRecordWriter(TaskAttemptContext job) throws IOException, InterruptedException {

        //将job对象传进去,才能方便对数据流进行操作
        MyRecordWriter recordWriter = new MyRecordWriter(job);

        return recordWriter;
    }
}

MyRecordWriter类

核心是重写write方法

public class MyRecordWriter extends RecordWriter<Text, NullWritable> {

    private FSDataOutputStream sxauOut;
    private FSDataOutputStream otherOut;

    public MyRecordWriter(TaskAttemptContext job) {
        //创建两个流
        try {
            FileSystem fs = FileSystem.get(job.getConfiguration());
            //输出路径
            sxauOut = fs.create(new Path("D:\\MapReduce_Data_Test\\myoutputformat\\sxau.log"));
            otherOut = fs.create(new Path("D:\\MapReduce_Data_Test\\myoutputformat\\other.log"));

        } catch (IOException e) {
            e.printStackTrace();
        }
    }

    @Override
    public void write(Text key, NullWritable value) throws IOException, InterruptedException {

        String log = key.toString();
        System.out.println(log);
        //具体的写出
        if (log.contains("sxau")){
            sxauOut.writeBytes(log+"\n");
        }else {
            otherOut.writeBytes(log+"\n");
        }

    }

    @Override
    public void close(TaskAttemptContext taskAttemptContext) throws IOException, InterruptedException {

        //关闭数据流
        IOUtils.closeStream(sxauOut);
        IOUtils.closeStream(otherOut);

    }
}

LogRunner类

设置输出格式

public class LogRunner extends Configured implements Tool {
    public static void main(String[] args) throws Exception {
        ToolRunner.run(new Configuration(),new LogRunner(),args);
    }

    @Override
    public int run(String[] args) throws Exception {
        //1.获取job
        Configuration conf = new Configuration();
        Job job = Job.getInstance(conf, "my outputFormat");

        //2.配置jar包路径
        job.setJarByClass(LogRunner.class);

        //3.关联mapper和reducer
        job.setMapperClass(LogMapper.class);
        job.setReducerClass(LogReducer.class);

        //4.设置map、reduce输出的k、v类型
        job.setMapOutputKeyClass(Text.class);
        job.setMapOutputValueClass(NullWritable.class);
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(NullWritable.class);

        //设置自定义输出格式
        job.setOutputFormatClass(MyOutputFormat.class);


        //5.设置统计文件输入的路径,将命令行的第一个参数作为输入文件的路径
        FileInputFormat.setInputPaths(job,new Path("D:\\MapReduce_Data_Test\\myoutputformat\\input"));
        //6.设置结果数据存放路径,将命令行的第二个参数作为数据的输出路径
        FileOutputFormat.setOutputPath(job,new Path("D:\\MapReduce_Data_Test\\myoutputformat\\output1"));
        return job.waitForCompletion(true) ? 0 : 1;//verbose：是否监控并打印job的信息
    }
}

运行结果