[hadoop@hadoop001 hadoop-2.6.0-cdh5.7.0]$ hadoop jar ./share/hadoop/mapreduce2/hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar pi 5 10[hadoop@hadoop001 hadoop-2.6.0-cdh5.7.0]$ hadoop jar ./share/hadoop/mapreduce2/hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar pi 5 10
Number of Maps = 5
Samples per Map = 10
19/02/24 12:54:02 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Wrote input for Map #0
Wrote input for Map #1
Wrote input for Map #2
Wrote input for Map #3
Wrote input for Map #4
Starting Job
19/02/24 12:54:05 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
19/02/24 12:54:05 INFO input.FileInputFormat: Total input paths to process : 5
19/02/24 12:54:05 INFO mapreduce.JobSubmitter: number of splits:5
19/02/24 12:54:06 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1550983677568_0001
19/02/24 12:54:06 INFO impl.YarnClientImpl: Submitted application application_1550983677568_0001
19/02/24 12:54:06 INFO mapreduce.Job: The url to track the job: http://hadoop001:8088/proxy/application_1550983677568_0001/
19/02/24 12:54:06 INFO mapreduce.Job: Running job: job_1550983677568_0001
19/02/24 12:54:12 INFO mapreduce.Job: Job job_1550983677568_0001 running in uber mode : false
19/02/24 12:54:12 INFO mapreduce.Job: map 0% reduce 0%
19/02/24 12:54:19 INFO mapreduce.Job: map 20% reduce 0%
19/02/24 12:54:21 INFO mapreduce.Job: map 40% reduce 0%
19/02/24 12:54:25 INFO mapreduce.Job: map 60% reduce 0%
19/02/24 12:54:26 INFO mapreduce.Job: map 80% reduce 0%
19/02/24 12:54:28 INFO mapreduce.Job: map 100% reduce 0%
19/02/24 12:54:29 INFO mapreduce.Job: map 100% reduce 100%
19/02/24 12:54:29 INFO mapreduce.Job: Job job_1550983677568_0001 completed successfully
19/02/24 12:54:29 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=116
FILE: Number of bytes written=670905
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=1330
HDFS: Number of bytes written=215
HDFS: Number of read operations=23
HDFS: Number of large read operations=0
HDFS: Number of write operations=3
Job Counters
Launched map tasks=5
Launched reduce tasks=1
Data-local map tasks=5
Total time spent by all maps in occupied slots (ms)=34068
Total time spent by all reduces in occupied slots (ms)=7920
Total time spent by all map tasks (ms)=34068
Total time spent by all reduce tasks (ms)=7920
Total vcore-seconds taken by all map tasks=34068
Total vcore-seconds taken by all reduce tasks=7920
Total megabyte-seconds taken by all map tasks=34885632
Total megabyte-seconds taken by all reduce tasks=8110080
Map-Reduce Framework
Map input records=5
Map output records=10
Map output bytes=90
Map output materialized bytes=140
Input split bytes=740
Combine input records=0
Combine output records=0
Reduce input groups=2
Reduce shuffle bytes=140
Reduce input records=10
Reduce output records=0
Spilled Records=20
Shuffled Maps =5
Failed Shuffles=0
Merged Map outputs=5
GC time elapsed (ms)=873
CPU time spent (ms)=2750
Physical memory (bytes) snapshot=1477246976
Virtual memory (bytes) snapshot=16641478656
Total committed heap usage (bytes)=1192755200
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=590
File Output Format Counters
Bytes Written=97
Job Finished in 24.098 seconds
Estimated value of Pi is 3.28000000000000000000
[hadoop@hadoop001 hadoop-2.6.0-cdh5.7.0]$
[hadoop@hadoop001 src_data]$ hdfs dfs -mkdir -p /wordcount/input[hadoop@hadoop001 src_data]$ hdfs dfs -ls /
19/02/24 13:02:33 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 4 items
drwxr-xr-x - hadoop supergroup 0 2019-02-23 19:17 /hello
drwx------ - hadoop supergroup 0 2019-02-24 12:54 /tmp
drwxr-xr-x - hadoop supergroup 0 2019-02-24 12:54 /user
drwxr-xr-x - hadoop supergroup 0 2019-02-24 13:02 /wordcount[hadoop@hadoop001 src_data]$ hdfs dfs -put a.log /wordcount/input
19/02/24 13:04:26 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@hadoop001 src_data]$ hdfs dfs -put b.txt /wordcount/input
19/02/24 13:04:35 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@hadoop001 src_data]$ hdfs dfs -ls /wordcount/input
19/02/24 13:04:49 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 2 items
-rw-r--r-- 1 hadoop supergroup 75 2019-02-24 13:04 /wordcount/input/a.log
-rw-r--r-- 1 hadoop supergroup 24 2019-02-24 13:04 /wordcount/input/b.txt[hadoop@hadoop001 hadoop-2.6.0-cdh5.7.0]$ hadoop jar ./share/hadoop/mapreduce2/hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar wordcount /wordcount/input /wordcount/output1
19/02/24 13:07:10 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
19/02/24 13:07:11 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
19/02/24 13:07:11 INFO input.FileInputFormat: Total input paths to process : 2
19/02/24 13:07:11 INFO mapreduce.JobSubmitter: number of splits:2
19/02/24 13:07:11 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1550983677568_0002
19/02/24 13:07:12 INFO impl.YarnClientImpl: Submitted application application_1550983677568_0002
19/02/24 13:07:12 INFO mapreduce.Job: The url to track the job: http://hadoop001:8088/proxy/application_1550983677568_0002/
19/02/24 13:07:12 INFO mapreduce.Job: Running job: job_1550983677568_0002
19/02/24 13:07:18 INFO mapreduce.Job: Job job_1550983677568_0002 running in uber mode : false
19/02/24 13:07:18 INFO mapreduce.Job: map 0% reduce 0%
19/02/24 13:07:23 INFO mapreduce.Job: map 50% reduce 0%
19/02/24 13:07:24 INFO mapreduce.Job: map 100% reduce 0%
19/02/24 13:07:28 INFO mapreduce.Job: map 100% reduce 100%
19/02/24 13:07:29 INFO mapreduce.Job: Job job_1550983677568_0002 completed successfully
19/02/24 13:07:29 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=211
FILE: Number of bytes written=334789
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=315
HDFS: Number of bytes written=105
HDFS: Number of read operations=9
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=2
Launched reduce tasks=1
Data-local map tasks=2
Total time spent by all maps in occupied slots (ms)=6033
Total time spent by all reduces in occupied slots (ms)=2644
Total time spent by all map tasks (ms)=6033
Total time spent by all reduce tasks (ms)=2644
Total vcore-seconds taken by all map tasks=6033
Total vcore-seconds taken by all reduce tasks=2644
Total megabyte-seconds taken by all map tasks=6177792
Total megabyte-seconds taken by all reduce tasks=2707456
Map-Reduce Framework
Map input records=13
Map output records=25
Map output bytes=199
Map output materialized bytes=217
Input split bytes=216
Combine input records=25
Combine output records=20
Reduce input groups=16
Reduce shuffle bytes=217
Reduce input records=20
Reduce output records=16
Spilled Records=40
Shuffled Maps =2
Failed Shuffles=0
Merged Map outputs=2
GC time elapsed (ms)=222
CPU time spent (ms)=1430
Physical memory (bytes) snapshot=681717760
Virtual memory (bytes) snapshot=8318640128
Total committed heap usage (bytes)=559939584
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=99
File Output Format Counters
Bytes Written=105
[hadoop@hadoop001 hadoop-2.6.0-cdh5.7.0]$
[hadoop@hadoop001 hadoop-2.6.0-cdh5.7.0]$ hadoop dfs -ls /wordcount/output1
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.
19/02/24 13:08:39 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 2 items
-rw-r--r-- 1 hadoop supergroup 0 2019-02-24 13:07 /wordcount/output1/_SUCCESS
-rw-r--r-- 1 hadoop supergroup 105 2019-02-24 13:07 /wordcount/output1/part-r-00000
[hadoop@hadoop001 hadoop-2.6.0-cdh5.7.0]$ hdfs dfs -text /wordcount/output1/part-r-00000
19/02/24 13:09:48 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
1 3
3 1
5 1
a 3
adai 1
b 3
c 2
d 1
dashu 1
e 1
f 1
fanren 1
jepon 1
jepson 1
ruoze 3
www.ruozedata.com 1
