A, MapReduce Overview
Hadoop MapReduce is a distributed computing framework for writing batch applications. Write a good program can be submitted to the Hadoop cluster for parallel processing of large data sets.
MapReduce job by the input data set into separate blocks, these blocks from the map
frame in a parallel manner to map
output queued, and then inputted to reduce
the. MapReduce framework specifically for <key,value>
key processing, input it as a set of job <key,value>
pair, and generates a set of <key,value>
serving as output. Output and output key
and value
must implement the Writable interface.
(input) <k1, v1> -> map -> <k2, v2> -> combine -> <k2, v2> -> reduce -> <k3, v3> (output)
Two, MapReduce programming model DESCRIPTION
Here with word frequency statistics as an example, MapReduce process flow is as follows:
-
the INPUT : read text files;
-
splitting : split the file according to the line, then the resulting
K1
number of lines,V1
represents text corresponding row; - Mapping : Each row in parallel on spaces split, split obtained
List(K2,V2)
, whereinK2
representing each word, since the frequency statistics is done, so thatV2
a value of 1, representing the 1 occurrence; - Shuffling : Since the
Mapping
operation may be on a different parallel processing machines, it is necessary byshuffling
the samekey
distribution data to the same value up a combined node, so as to count the final result obtained at this timeK2
for each wordList(V2)
to be iterative collectionV2
is Mapping of V2; - Reducing : Case Here is the total number of statistical word appears, it is
Reducing
toList(V2)
be summed reduction operation, the final output.
MapReduce programming model splitting
and shuffing
operations are implemented by the framework, we need only realize their own programming mapping
and reducing
, which is the source of the title of MapReduce.
三, combiner & Partitions
3.1 InputFormat & RecordReaders
InputFormat
File into a plurality of output InputSplit
by RecordReaders
the InputSplit
conversion to the standard <key, value> key-value pairs, as the output of the map. This is only the first step in the sense that logically split and converted to a standard key-value pair format, in order to more map
to provide input for parallel processing.
3.2 Combiner
combiner
Is map
optional post-operation, it is actually a localized reduce
operation, it is mainly in map
the intermediate file is calculated to make a simple merge duplicate key
operation value. Here with word frequency statistics, for example:
map
在遇到一个 hadoop 的单词时就会记录为 1,但是这篇文章里 hadoop 可能会出现 n 多次,那么 map
输出文件冗余就会很多,因此在 reduce
计算前对相同的 key 做一个合并操作,那么需要传输的数据量就会减少,传输效率就可以得到提升。
但并非所有场景都适合使用 combiner
,使用它的原则是 combiner
的输出不会影响到 reduce
计算的最终输入,例如:求总数,最大值,最小值时都可以使用 combiner
,但是做平均值计算则不能使用 combiner
。
不使用 combiner 的情况:
<div align="center"> <img width="600px" src="https://raw.githubusercontent.com/heibaiying/BigData-Notes/master/pictures/mapreduce-without-combiners.png"/>; </div>
使用 combiner 的情况:
<div align="center"> <img width="600px" src="https://raw.githubusercontent.com/heibaiying/BigData-Notes/master/pictures/mapreduce-with-combiners.png"/>; </div>
可以看到使用 combiner 的时候,需要传输到 reducer 中的数据由 12keys,降低到 10keys。降低的幅度取决于你 keys 的重复率,下文词频统计案例会演示用 combiner 降低数百倍的传输量。
3.3 Partitioner
partitioner
可以理解成分类器,将 map
的输出按照 key 值的不同分别分给对应的 reducer
,支持自定义实现,下文案例会给出演示。
四、MapReduce词频统计案例
4.1 项目简介
这里给出一个经典的词频统计的案例:统计如下样本数据中每个单词出现的次数。
Spark HBase
Hive Flink Storm Hadoop HBase Spark
Flink
HBase Storm
HBase Hadoop Hive Flink
HBase Flink Hive Storm
Hive Flink Hadoop
HBase Hive
Hadoop Spark HBase Storm
HBase Hadoop Hive Flink
HBase Flink Hive Storm
Hive Flink Hadoop
HBase Hive
为方便大家开发,我在项目源码中放置了一个工具类 WordCountDataUtils
,用于模拟产生词频统计的样本,生成的文件支持输出到本地或者直接写到 HDFS 上。
项目代码下载地址:hadoop-word-count
4.2 项目依赖
想要进行 MapReduce 编程,需要导入 hadoop-client
依赖:
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${hadoop.version}</version>
</dependency>
4.3 WordCountMapper
将每行数据按照指定分隔符进行拆分。这里需要注意在 MapReduce 中必须使用 Hadoop 定义的类型,因为 Hadoop 预定义的类型都是可序列化,可比较的,所有类型均实现了 WritableComparable
接口。
public class WordCountMapper extends Mapper<LongWritable, Text, Text, IntWritable> {
@Override
protected void map(LongWritable key, Text value, Context context) throws IOException,
InterruptedException {
String[] words = value.toString().split("\t");
for (String word : words) {
context.write(new Text(word), new IntWritable(1));
}
}
}
WordCountMapper
对应下图的 Mapping 操作:
WordCountMapper
继承自 Mappe
类,这是一个泛型类,定义如下:
WordCountMapper extends Mapper<LongWritable, Text, Text, IntWritable>
public class Mapper<KEYIN, VALUEIN, KEYOUT, VALUEOUT> {
......
}
- KEYIN :
mapping
输入 key 的类型,即每行的偏移量 (每行第一个字符在整个文本中的位置),Long
类型,对应 Hadoop 中的LongWritable
类型; - VALUEIN :
mapping
输入 value 的类型,即每行数据;String
类型,对应 Hadoop 中Text
类型; - KEYOUT :
mapping
输出的 key 的类型,即每个单词;String
类型,对应 Hadoop 中Text
类型; - VALUEOUT:
mapping
输出 value 的类型,即每个单词出现的次数;这里用int
类型,对应IntWritable
类型。
4.4 WordCountReducer
在 Reduce 中进行单词出现次数的统计:
public class WordCountReducer extends Reducer<Text, IntWritable, Text, IntWritable> {
@Override
protected void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException,
InterruptedException {
int count = 0;
for (IntWritable value : values) {
count += value.get();
}
context.write(key, new IntWritable(count));
}
}
如下图,shuffling
的输出是 reduce 的输入。这里的 key 是每个单词,values 是一个可迭代的数据类型,类似 (1,1,1,...)
。
4.4 WordCountApp
组装 MapReduce 作业,并提交到服务器运行,代码如下:
/**
* 组装作业 并提交到集群运行
*/
public class WordCountApp {
// 这里为了直观显示参数 使用了硬编码,实际开发中可以通过外部传参
private static final String HDFS_URL = "hdfs://192.168.0.107:8020";
private static final String HADOOP_USER_NAME = "root";
public static void main(String[] args) throws Exception {
// 文件输入路径和输出路径由外部传参指定
if (args.length < 2) {
System.out.println("Input and output paths are necessary!");
return;
}
// 需要指明 hadoop 用户名,否则在 HDFS 上创建目录时可能会抛出权限不足的异常
System.setProperty("HADOOP_USER_NAME", HADOOP_USER_NAME);
Configuration configuration = new Configuration();
// 指明 HDFS 的地址
configuration.set("fs.defaultFS", HDFS_URL);
// 创建一个 Job
Job job = Job.getInstance(configuration);
// 设置运行的主类
job.setJarByClass(WordCountApp.class);
// 设置 Mapper 和 Reducer
job.setMapperClass(WordCountMapper.class);
job.setReducerClass(WordCountReducer.class);
// 设置 Mapper 输出 key 和 value 的类型
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(IntWritable.class);
// 设置 Reducer 输出 key 和 value 的类型
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
// 如果输出目录已经存在,则必须先删除,否则重复运行程序时会抛出异常
FileSystem fileSystem = FileSystem.get(new URI(HDFS_URL), configuration, HADOOP_USER_NAME);
Path outputPath = new Path(args[1]);
if (fileSystem.exists(outputPath)) {
fileSystem.delete(outputPath, true);
}
// 设置作业输入文件和输出文件的路径
FileInputFormat.setInputPaths(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, outputPath);
// 将作业提交到群集并等待它完成,参数设置为 true 代表打印显示对应的进度
boolean result = job.waitForCompletion(true);
// 关闭之前创建的 fileSystem
fileSystem.close();
// 根据作业结果,终止当前运行的 Java 虚拟机,退出程序
System.exit(result ? 0 : -1);
}
}
需要注意的是:如果不设置 Mapper
操作的输出类型,则程序默认它和 Reducer
操作输出的类型相同。
4.5 提交到服务器运行
在实际开发中,可以在本机配置 hadoop 开发环境,直接在 IDE 中启动进行测试。这里主要介绍一下打包提交到服务器运行。由于本项目没有使用除 Hadoop 外的第三方依赖,直接打包即可:
# mvn clean package
使用以下命令提交作业:
hadoop jar /usr/appjar/hadoop-word-count-1.0.jar \
com.heibaiying.WordCountApp \
/wordcount/input.txt /wordcount/output/WordCountApp
作业完成后查看 HDFS 上生成目录:
# 查看目录
hadoop fs -ls /wordcount/output/WordCountApp
# 查看统计结果
hadoop fs -cat /wordcount/output/WordCountApp/part-r-00000
五、词频统计案例进阶之Combiner
5.1 代码实现
想要使用 combiner
功能只要在组装作业时,添加下面一行代码即可:
// 设置 Combiner
job.setCombinerClass(WordCountReducer.class);
5.2 执行结果
加入 combiner
后统计结果是不会有变化的,但是可以从打印的日志看出 combiner
的效果:
Not a member of combiner
the print log:
Adding combiner
print log after follows:
Here we have only one input file and less than 128M, so only a Map for processing. After combiner can be seen, the Records 3519
reduced to 6
(sample species only six kinds of words), the amount of data used in this embodiment can greatly reduce the need combiner transmitted.
Six, word frequency statistics advanced cases of Partitioner
6.1 The default Partitioner
It is assumed that there is a demand: the output statistics of different words to different files. This demand is actually more common, while sales of products such as statistics, the results need to be split according to product categories. To achieve this function, you need to use custom Partitioner
.
Here we introduce MapReduce default classification rules: in building job time, if not specified, the default used is HashPartitioner
: hash hash value of the key and numReduceTasks
take the remainder. Its implementation is as follows:
public class HashPartitioner<K, V> extends Partitioner<K, V> {
public int getPartition(K key, V value,
int numReduceTasks) {
return (key.hashCode() & Integer.MAX_VALUE) % numReduceTasks;
}
}
6.2 custom Partitioner
Here we inherit Partitioner
custom classification rules, here are classified in accordance with the words:
public class CustomPartitioner extends Partitioner<Text, IntWritable> {
public int getPartition(Text text, IntWritable intWritable, int numPartitions) {
return WordCountDataUtils.WORD_LIST.indexOf(text.toString());
}
}
In the construction of job
the specified classification rules using our own time, and set reduce
the number of:
// 设置自定义分区规则
job.setPartitionerClass(CustomPartitioner.class);
// 设置 reduce 个数
job.setNumReduceTasks(WordCountDataUtils.WORD_LIST.size());
6.3 execution results
Results of the following, six files are generated, the statistics for each file of the corresponding word:
Reference material
- MapReduce distributed computing framework
- Apache Hadoop 2.9.2 > MapReduce Tutorial
- MapReduce - Combiners
More big data series can be found GitHub open source project : Big Data Getting Started