Mapreduce two ways to run programs on hadoop

Before learning the knowledge of hadoop period of time to learn the theoretical basis of the actual operation at the same time it can be more skilled, ado it said to run on hadoop one of the most simple words count program

First of all I would like to paste the program's source code for your reference code is divided into three parts written Run, map phase, reduce stage

Map：

  
  
package wordsCount;

import java.io.IOException;
import java.util.StringTokenizer;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;

public class WordsMapper extends Mapper<LongWritable, Text, Text, IntWritable>{
@Override
protected void map(LongWritable key, Text value, Mapper<LongWritable, Text, Text, IntWritable>.Context context)
throws IOException, InterruptedException {

String line = value.toString();
StringTokenizer st = new StringTokenizer(line);
while(st.hasMoreTokens()){
String word = st.nextToken();
context.write( new Text(word), new IntWritable( 1));
}

}

}
  
  

Reduce：

  
  
package wordsCount;

import java.io.IOException;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;

public class WordsReduce extends Reducer<Text, IntWritable, Text, IntWritable>{

@Override
protected void reduce(Text key, Iterable<IntWritable> iterator,
Reducer<Text, IntWritable, Text, IntWritable>.Context context) throws IOException, InterruptedException {
// TODO method of automatically generating stubs
int I = 0 ;
for(IntWritable i:iterator){
sum = sum + i.get();
}
context.write(key, new IntWritable(sum));
}
}
  
  

Run：

  
  
package wordsCount;


import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;

public class Run {

public static void main(String[] args) throws Exception{
// TODO method of automatically generating stubs
Configuration configuration = new Configuration();
Job job = new Job(configuration);
job.setJarByClass(Run.class);
job.setJobName( "words count!");

job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);

job.setInputFormatClass(TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);

job.setMapperClass(WordsMapper.class);
job.setReducerClass(WordsReduce.class);

FileInputFormat.addInputPath(job, new Path( "hdfs://192.168.1.111:9000/user/input/wc/"));
FileOutputFormat.setOutputPath(job, new Path( "hdfs://192.168.1.111:9000/user/result/"));

job.waitForCompletion( true);
}
}
  
  

Run inside the input and output paths according to their modified to

This program will not have to explain it can be found everywhere

First run on hadoop this program in two ways

Method One: to compile their own software and connected hadoop (I use MyEclipse to link hadoop), run the program directly. MyEclipse tutorials will be connected hadoop I will give a link for your reference at the end of the article.

See the following message it means you are successful then you then your output folder which will be able to view the results of the operation

The second file which is the content output

The second method: the mapreduce packaged into a jar file

Here a brief speech method of packaging

Then the next step can be completed

On the packaged jar files to your install hadoop machine (my hadoop cluster is installed in the linux virtual machine) after using SSH pass over the jar:

Hadoop executable file in the bin directory under your installation directory hadoop, and then do the following on it:

Under the statement to explain my shell

/home/xiaohuihui/wordscount.jar: the location of the jar file after the package (spread virtual machine location)

wordsCount / Run: The name of the place your jar package main function (main function here is Run.class) can open your jar file viewer will know

You can also add a file input and output path, but that I have set up after this statement in my program

See the following output after the shell if you run the above statement, then congratulations to you, success! !

You can see the results in your Eclipse connected hadoop view, it can also hdfs page to view the file system (localhost: 50070).

There is also a very important step is, before running to ensure your hadoop has started, you can to see if your cluster hadoop process has been started by jps

Eclipse连接hadoop：http://blog.csdn.net/xjavasunjava/article/details/12320045