Hadoop,Spark ：文件输出压缩

Spark最终结果压缩

sparkContext.textFile(inputPath).flatMap(x => {
    TestLog.readFromString(x)
}).saveAsTextFile(outputPath, classOf[GzipCodec])

• 1
• 2
• 3

MapReduce最终结果的压缩

FileOutputFormat.setCompressOutput(job, true);//设置压缩
FileOutputFormat.setOutputCompressorClass(job, GzipCodec.class);//压缩类型

• 1
• 2

Mapper 中间结果压缩

Configuration conf = new Configuration();
conf.setBoolean("mapred.compress.map.output",true);
conf.setClass("mapred.map.output.compression.codec",GzipCodec.class,CompressionCodec.class);