2.7.3 version of hadoop:
The directory where the jar program is located: $HADOOP_HOME/shar/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar
1. Create the test text locally:
mkdir /home/hadoop/data // Create data folder touch testinput.txt // Create test text vim testinput.txt // Modify text // Add a line to the text this is a test log cat testinput.txt // check txt text
2.hdfs
hadoop fs - ls / // View the directory on hdfs hadoop fs - mkdir /input // Create an input directory hadoop fs - rm -r /output // If there is an output directory, delete hadoop fs -put /home/hadoop/data/ testiinput.txt /input // Upload the test text to the input directory
hadoop jar /home/hadoop/softwares/hadoop- 2.7 . 3 /share/hadoop/mapreduce/hadoop-mapreduce-examples- 2.7 . 3 .jar wordcount /input /output
// execute the program, wordcount is the main class name of the program, /input input directory/output output directory (the output directory cannot exist)
hadoop fs - ls /output // check the output directory after completion hadoop fs - cat /output/part-r- 00000 // View the output result
working process:
18/04/16 19:43:15 INFO client.RMProxy: Connecting to ResourceManager at hadoop/192.168.30.129:8032 18/04/16 19:43:18 INFO input.FileInputFormat: Total input paths to process : 1 18/04/16 19:43:18 INFO mapreduce.JobSubmitter: number of splits:1 18/04/16 19:43:18 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1523884458275_0001 18/04/16 19:43:19 INFO impl.YarnClientImpl: Submitted application application_1523884458275_0001 18/04/16 19:43:19 INFO mapreduce.Job: The url to track the job: http://hadoop:8088/proxy/application_1523884458275_0001/ 18/04/16 19:43:19 INFO mapreduce.Job: Running job: job_1523884458275_0001 18/04/16 19:43:41 INFO mapreduce.Job: Job job_1523884458275_0001 running in uber mode : false 18/04/16 19:43:41 INFO mapreduce.Job: map 0% reduce 0% 18/04/16 19:43:51 INFO mapreduce.Job: map 100% reduce 0% 18/04/16 19:44:03 INFO mapreduce.Job: map 100% reduce 100% 18/04/16 19:44:05 INFO mapreduce.Job: Job job_1523884458275_0001 completed successfully 18/04/16 19:44:05 INFO mapreduce.Job: Counters: 49 File System Counters FILE: Number of bytes read=6 FILE: Number of bytes written=237375 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=100 HDFS: Number of bytes written=0 HDFS: Number of read operations=6 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 Job Counters Launched map tasks=1 Launched reduce tasks=1 Other local map tasks=1 Total time spent by all maps in occupied slots (ms)=8459 Total time spent by all reduces in occupied slots (ms)=8037 Total time spent by all map tasks (ms)=8459 Total time spent by all reduce tasks (ms)=8037 Total vcore-milliseconds taken by all map tasks=8459 Total vcore-milliseconds taken by all reduce tasks=8037 Total megabyte-milliseconds taken by all map tasks=8662016 Total megabyte-milliseconds taken by all reduce tasks=8229888 Map-Reduce Framework Map input records=0 Map output records=0 Map output bytes=0 Map output materialized bytes=6 Input split bytes=100 Combine input records=0 Combine output records=0 Reduce input groups=0 Reduce shuffle bytes=6 Reduce input records=0 Reduce output records=0 Spilled Records=0 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=264 CPU time spent (ms)=1460 Physical memory (bytes) snapshot=287879168 Virtual memory (bytes) snapshot=3887841280 Total committed heap usage (bytes)=139845632 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=0 File Output Format Counters Bytes Written=0