In order to estimate the time required for the cluster to read and write files in the future, the cluster read and write performance can be tested
1) Test HDFS write performance
Test content: Write 10 128M files to HDFS cluster
[atguigu@hadoop102 mapreduce]$ hadoop jar /opt/module/hadoop-2.7.2/share/hadoop/mapreduce/hadoop-mapreduce-client- jobclient-2.7.2-tests.jar TestDFSIO -write -nrFiles 10 -fileSize 128MB
2) Test HDFS read performance
Test content: read 10 128M files in HDFS cluster
[atguigu@hadoop102 mapreduce]$ hadoop jar /opt/module/hadoop-2.7.2/share/hadoop/mapreduce/hadoop-mapreduce-client- jobclient-2.7.2-tests.jar TestDFSIO -read -nrFiles 10 -fileSize 128MB
3) Delete test generated data
[atguigu@hadoop102 mapreduce]$ hadoop jar /opt/module/hadoop-2.7.2/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.2-tests.jar TestDFSIO -clean
4) Use the Sort program to evaluate MapReduce
(1) Use RandomWriter to generate random numbers, each node runs 10 Map tasks, and each Map generates a binary random number about 1G in size
[atguigu@hadoop102 mapreduce]$ hadoop jar /opt/module /hadoop-2.7.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar randomwriter random-data
(2) Execute Sort program
[atguigu@hadoop102 mapreduce]$ hadoop jar /opt/module/hadoop- 2.7.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar sort random-data sorted-data
(3) Verify that the data is really sorted
[atguigu@hadoop102 mapreduce]$ hadoop jar /opt /module/hadoop-2.7.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar testmapredsort -sortInput random-data -sortOutput sorted-data