为了毕业，我使用一台腾讯云的服务器部署hadoop进行开发。
系统：CentOS Linux release 7.7.1908 (Core)
hadoop版本：hadoop-3.1.2.tar.gz
1、首先，我们必须要有java环境，java版本1.8（向上兼容到了j10）。同时要将java配置进环境变量中
[root@shengxi ~]# java -version
openjdk version "1.8.0_222"
OpenJDK Runtime Environment (build 1.8.0_222-b10)
OpenJDK 64-Bit Server VM (build 25.222-b10, mixed mode)
[root@shengxi ~]# vim /etc/profile
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.222/
export JRE_HOME=$JAVA_HOME/jre
export CLASSPATH=$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH
export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin
2、配置本机虚拟域名。因为我是云服务器，所以不应该使用127.0.0.1进行hadoop配置的，所以应该使用本机ip（注意：不是公网ip，而是内网ip）。
//查看ip
[root@shengxi ~]# ifconfig -a
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.17.x.x netmask 255.255.240.0 broadcast 172.17.15.255
ether 52:54:00:8a:fa:12 txqueuelen 1000 (Ethernet)
RX packets 1767653 bytes 1932785383 (1.8 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 775608 bytes 93677869 (89.3 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
loop txqueuelen 1 (Local Loopback)
RX packets 2 bytes 276 (276.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2 bytes 276 (276.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
其中eth0中的172.17.x.x就是我的本地ip了。
[root@shengxi ~]# vim /etc/hosts
172.17.x.x shengxi
172.17.x.x hadoop
127.0.0.1 localhost.localdomain localhost
127.0.0.1 localhost4.localdomain4 localhost4
::1 localhost.localdomain localhost
::1 localhost6.localdomain6 localhost6
增加一句 ip: hostname来配置虚拟域名，这里配置了一个hadoop作为hadoop开发用的虚拟域名。
3、增加一个hadoop用户，因为使用root用户启动的hadoop会出现一些报错，而且访问的时候会出现路径不对应的情况。因此我直接增加一个hadoop user配置root权限。（可以直接将/etc/passwd中的对应用户的uid改为0——root权限)
//新增用户
[root@shengxi ~]# adduser hadoop
//修改密码
[root@shengxi ~]# passwd hadoop
Changing password for user hadoop.
//输入两个新密码
New password:
Retype new password:
passwd: all authentication tokens updated successfully.修改/etc/sudoers文件，增加一句hadoop ALL=(ALL) NOPASSWD:ALL这样hadoop使用root权限时就不用输入密码了。
## Allow root to run any commands anywhere
root ALL=(ALL) ALL
hadoop ALL=(ALL) NOPASSWD:ALL3、ssh安装，因为我们是云服务器，自带ssh环境，就不需要安装了。直接配置不需要密码登录就好了。
4、获取压缩包，并解压。
//获取压缩包
[root@shengxi ~]# wget https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/hadoop-3.1.2/hadoop-3.1.2.tar.gz
//解压
[root@shengxi ~]# tar -zxvf hadoop-3.1.2.tar.gz
5、为了方便管理，将hadoop文件夹移动到一个你习惯的文件路径下，然后将文件夹所有人改成hadoop。
[root@shengxi ~]# mv hadoop-3.1.2 /usr/local/
[root@shengxi ~]# cd /usr/local/
[root@shengxi local]# ls
bin games include lib64 qcloud share yd.socket.server
etc hadoop-3.1.2 lib libexec sbin src
[root@shengxi local]#
修改文件所属用户和用户组
[root@shengxi local]# chown -R hadoop:root hadoop-3.1.2/
[root@shengxi local]# ls -l hadoop-3.1.2/
total 204
drwxrwxrwx 2 hadoop root 4096 Jan 29 2019 bin
drwxrwxrwx 3 hadoop root 4096 Jan 29 2019 etc
drwxrwxrwx 2 hadoop root 4096 Jan 29 2019 include
drwxrwxrwx 3 hadoop root 4096 Jan 29 2019 lib
drwxrwxrwx 4 hadoop root 4096 Jan 29 2019 libexec
-rwxrwxrwx 1 hadoop root 147145 Jan 23 2019 LICENSE.txt
-rwxrwxrwx 1 hadoop root 21867 Jan 23 2019 NOTICE.txt
-rwxrwxrwx 1 hadoop root 1366 Jan 23 2019 README.txt
drwxrwxrwx 3 hadoop root 4096 Jan 29 2019 sbin
drwxrwxrwx 4 hadoop root 4096 Jan 29 2019 share
配置环境变量
[hadoop@shengxi ~]$ vim /etc/profile
#配置hadoop环境变量
export HADOOP_HOME=/usr/local/hadoop-3.1.2
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_YARN_HOME=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop
[hadoop@shengxi ~]$ source /etc/profile
6、测试单机hadoop是否成功。
*测试hadoop version
[hadoop@shengxi ~]$ source /etc/profile
[hadoop@shengxi ~]$ hadoop version
Hadoop 3.1.2
Source code repository https://github.com/apache/hadoop.git -r 1019dde65bcf12e05ef48ac71e84550d589e5d9a
Compiled by sunilg on 2019-01-29T01:39Z
Compiled with protoc 2.5.0
From source with checksum 64b8bdd4ca6e77cce75a93eb09ab2a9
This command was run using /usr/local/hadoop-3.1.2/share/hadoop/common/hadoop-common-3.1.2.jar
[hadoop@shengxi ~]$
**测试二，使用mapreduce统计单词出现的次数。
在用户文件夹下创建一个input文件夹，在里面写几个文件。我写了三个txt，每个文件间有重复的单词。
[hadoop@shengxi ~]$ mkdir input
[hadoop@shengxi ~]$ cd input
//新建并编辑三个文件
[hadoop@shengxi input]$ vim f1.txt
[hadoop@shengxi input]$ vim f2.txt
[hadoop@shengxi input]$ vim f3.txt
[hadoop@shengxi input]$ ll
total 12
-rw-r--r-- 1 root root 11 Oct 13 13:59 f1.txt
-rw-r--r-- 1 root root 25 Oct 13 14:01 f2.txt
-rw-r--r-- 1 root root 19 Oct 13 14:01 f3.txt
//调用方法 注意：不可以创建输出文件夹，如果存在output文件夹，那就改结果路径，或者删除output
hadoop jar /usr/local/hadoop-3.1.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.2.jar wordcount input output
2019-10-13 14:09:28,762 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2019-10-13 14:09:28,879 INFO impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s).
2019-10-13 14:09:28,879 INFO impl.MetricsSystemImpl: JobTracker metrics system started
2019-10-13 14:09:29,089 INFO input.FileInputFormat: Total input files to process : 3
2019-10-13 14:09:29,116 INFO mapreduce.JobSubmitter: number of splits:3
2019-10-13 14:09:29,345 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local696442689_0001
2019-10-13 14:09:29,346 INFO mapreduce.JobSubmitter: Executing with tokens: []
2019-10-13 14:09:29,569 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
2019-10-13 14:09:29,570 INFO mapreduce.Job: Running job: job_local696442689_0001
2019-10-13 14:09:29,575 INFO mapred.LocalJobRunner: OutputCommitter set in config null
2019-10-13 14:09:29,583 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2019-10-13 14:09:29,583 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2019-10-13 14:09:29,583 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2019-10-13 14:09:29,650 INFO mapred.LocalJobRunner: Waiting for map tasks
2019-10-13 14:09:29,650 INFO mapred.LocalJobRunner: Starting task: attempt_local696442689_0001_m_000000_0
2019-10-13 14:09:29,672 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2019-10-13 14:09:29,672 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2019-10-13 14:09:29,691 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2019-10-13 14:09:29,695 INFO mapred.MapTask: Processing split: file:/home/hadoop/input/f2.txt:0+25
2019-10-13 14:09:29,802 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2019-10-13 14:09:29,802 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2019-10-13 14:09:29,802 INFO mapred.MapTask: soft limit at 83886080
2019-10-13 14:09:29,802 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2019-10-13 14:09:29,803 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2019-10-13 14:09:29,810 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2019-10-13 14:09:29,815 INFO mapred.LocalJobRunner:
2019-10-13 14:09:29,815 INFO mapred.MapTask: Starting flush of map output
2019-10-13 14:09:29,815 INFO mapred.MapTask: Spilling map output
2019-10-13 14:09:29,815 INFO mapred.MapTask: bufstart = 0; bufend = 42; bufvoid = 104857600
2019-10-13 14:09:29,816 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214384(104857536); length = 13/6553600
2019-10-13 14:09:29,827 INFO mapred.MapTask: Finished spill 0
2019-10-13 14:09:29,835 INFO mapred.Task: Task:attempt_local696442689_0001_m_000000_0 is done. And is in the process of committing
2019-10-13 14:09:29,846 INFO mapred.LocalJobRunner: map
2019-10-13 14:09:29,846 INFO mapred.Task: Task 'attempt_local696442689_0001_m_000000_0' done.
2019-10-13 14:09:29,853 INFO mapred.Task: Final Counters for attempt_local696442689_0001_m_000000_0: Counters: 18
File System Counters
FILE: Number of bytes read=316771
FILE: Number of bytes written=815692
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
Map-Reduce Framework
Map input records=1
Map output records=4
Map output bytes=42
Map output materialized bytes=56
Input split bytes=95
Combine input records=4
Combine output records=4
Spilled Records=4
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=22
Total committed heap usage (bytes)=135335936
File Input Format Counters
Bytes Read=25
2019-10-13 14:09:29,853 INFO mapred.LocalJobRunner: Finishing task: attempt_local696442689_0001_m_000000_0
2019-10-13 14:09:29,853 INFO mapred.LocalJobRunner: Starting task: attempt_local696442689_0001_m_000001_0
2019-10-13 14:09:29,858 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2019-10-13 14:09:29,859 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2019-10-13 14:09:29,859 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2019-10-13 14:09:29,860 INFO mapred.MapTask: Processing split: file:/home/hadoop/input/f3.txt:0+19
2019-10-13 14:09:29,906 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2019-10-13 14:09:29,906 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2019-10-13 14:09:29,906 INFO mapred.MapTask: soft limit at 83886080
2019-10-13 14:09:29,906 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2019-10-13 14:09:29,906 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2019-10-13 14:09:29,908 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2019-10-13 14:09:29,909 INFO mapred.LocalJobRunner:
2019-10-13 14:09:29,909 INFO mapred.MapTask: Starting flush of map output
2019-10-13 14:09:29,909 INFO mapred.MapTask: Spilling map output
2019-10-13 14:09:29,909 INFO mapred.MapTask: bufstart = 0; bufend = 32; bufvoid = 104857600
2019-10-13 14:09:29,909 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214388(104857552); length = 9/6553600
2019-10-13 14:09:29,910 INFO mapred.MapTask: Finished spill 0
2019-10-13 14:09:29,923 INFO mapred.Task: Task:attempt_local696442689_0001_m_000001_0 is done. And is in the process of committing
2019-10-13 14:09:29,924 INFO mapred.LocalJobRunner: map
2019-10-13 14:09:29,924 INFO mapred.Task: Task 'attempt_local696442689_0001_m_000001_0' done.
2019-10-13 14:09:29,925 INFO mapred.Task: Final Counters for attempt_local696442689_0001_m_000001_0: Counters: 18
File System Counters
FILE: Number of bytes read=317094
FILE: Number of bytes written=815768
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
Map-Reduce Framework
Map input records=1
Map output records=3
Map output bytes=32
Map output materialized bytes=44
Input split bytes=95
Combine input records=3
Combine output records=3
Spilled Records=3
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=21
Total committed heap usage (bytes)=182521856
File Input Format Counters
Bytes Read=19
2019-10-13 14:09:29,925 INFO mapred.LocalJobRunner: Finishing task: attempt_local696442689_0001_m_000001_0
2019-10-13 14:09:29,925 INFO mapred.LocalJobRunner: Starting task: attempt_local696442689_0001_m_000002_0
2019-10-13 14:09:29,935 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2019-10-13 14:09:29,935 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2019-10-13 14:09:29,935 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2019-10-13 14:09:29,936 INFO mapred.MapTask: Processing split: file:/home/hadoop/input/f1.txt:0+11
2019-10-13 14:09:29,979 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
2019-10-13 14:09:29,979 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
2019-10-13 14:09:29,979 INFO mapred.MapTask: soft limit at 83886080
2019-10-13 14:09:29,979 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
2019-10-13 14:09:29,979 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
2019-10-13 14:09:29,981 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2019-10-13 14:09:29,982 INFO mapred.LocalJobRunner:
2019-10-13 14:09:29,982 INFO mapred.MapTask: Starting flush of map output
2019-10-13 14:09:29,983 INFO mapred.MapTask: Spilling map output
2019-10-13 14:09:29,983 INFO mapred.MapTask: bufstart = 0; bufend = 20; bufvoid = 104857600
2019-10-13 14:09:29,983 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214392(104857568); length = 5/6553600
2019-10-13 14:09:29,984 INFO mapred.MapTask: Finished spill 0
2019-10-13 14:09:29,996 INFO mapred.Task: Task:attempt_local696442689_0001_m_000002_0 is done. And is in the process of committing
2019-10-13 14:09:30,000 INFO mapred.LocalJobRunner: map
2019-10-13 14:09:30,000 INFO mapred.Task: Task 'attempt_local696442689_0001_m_000002_0' done.
2019-10-13 14:09:30,001 INFO mapred.Task: Final Counters for attempt_local696442689_0001_m_000002_0: Counters: 18
File System Counters
FILE: Number of bytes read=317409
FILE: Number of bytes written=815830
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
Map-Reduce Framework
Map input records=1
Map output records=2
Map output bytes=20
Map output materialized bytes=30
Input split bytes=95
Combine input records=2
Combine output records=2
Spilled Records=2
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=25
Total committed heap usage (bytes)=168112128
File Input Format Counters
Bytes Read=11
2019-10-13 14:09:30,001 INFO mapred.LocalJobRunner: Finishing task: attempt_local696442689_0001_m_000002_0
2019-10-13 14:09:30,001 INFO mapred.LocalJobRunner: map task executor complete.
2019-10-13 14:09:30,007 INFO mapred.LocalJobRunner: Waiting for reduce tasks
2019-10-13 14:09:30,007 INFO mapred.LocalJobRunner: Starting task: attempt_local696442689_0001_r_000000_0
2019-10-13 14:09:30,034 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 2
2019-10-13 14:09:30,034 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
2019-10-13 14:09:30,035 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ]
2019-10-13 14:09:30,037 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@17c3dca2
2019-10-13 14:09:30,038 WARN impl.MetricsSystemImpl: JobTracker metrics system already initialized!
2019-10-13 14:09:30,064 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=326402048, maxSingleShuffleLimit=81600512, mergeThreshold=215425360, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2019-10-13 14:09:30,078 INFO reduce.EventFetcher: attempt_local696442689_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2019-10-13 14:09:30,103 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local696442689_0001_m_000001_0 decomp: 40 len: 44 to MEMORY
2019-10-13 14:09:30,115 INFO reduce.InMemoryMapOutput: Read 40 bytes from map-output for attempt_local696442689_0001_m_000001_0
2019-10-13 14:09:30,116 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 40, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->40
2019-10-13 14:09:30,118 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local696442689_0001_m_000002_0 decomp: 26 len: 30 to MEMORY
2019-10-13 14:09:30,120 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:270)
at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:147)
at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:208)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2019-10-13 14:09:30,122 INFO reduce.InMemoryMapOutput: Read 26 bytes from map-output for attempt_local696442689_0001_m_000002_0
2019-10-13 14:09:30,122 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 26, inMemoryMapOutputs.size() -> 2, commitMemory -> 40, usedMemory ->66
2019-10-13 14:09:30,124 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local696442689_0001_m_000000_0 decomp: 52 len: 56 to MEMORY
2019-10-13 14:09:30,125 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:270)
at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:147)
at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:208)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2019-10-13 14:09:30,125 INFO reduce.InMemoryMapOutput: Read 52 bytes from map-output for attempt_local696442689_0001_m_000000_0
2019-10-13 14:09:30,125 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 52, inMemoryMapOutputs.size() -> 3, commitMemory -> 66, usedMemory ->118
2019-10-13 14:09:30,126 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
2019-10-13 14:09:30,127 INFO mapred.LocalJobRunner: 3 / 3 copied.
2019-10-13 14:09:30,127 INFO reduce.MergeManagerImpl: finalMerge called with 3 in-memory map-outputs and 0 on-disk map-outputs
2019-10-13 14:09:30,131 INFO mapred.Merger: Merging 3 sorted segments
2019-10-13 14:09:30,131 INFO mapred.Merger: Down to the last merge-pass, with 3 segments left of total size: 91 bytes
2019-10-13 14:09:30,132 INFO reduce.MergeManagerImpl: Merged 3 segments, 118 bytes to disk to satisfy reduce memory limit
2019-10-13 14:09:30,132 INFO reduce.MergeManagerImpl: Merging 1 files, 118 bytes from disk
2019-10-13 14:09:30,133 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
2019-10-13 14:09:30,133 INFO mapred.Merger: Merging 1 sorted segments
2019-10-13 14:09:30,137 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptor
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)
at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:270)
at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:147)
at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:208)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2019-10-13 14:09:30,138 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 106 bytes
2019-10-13 14:09:30,138 INFO mapred.LocalJobRunner: 3 / 3 copied.
2019-10-13 14:09:30,140 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
2019-10-13 14:09:30,141 INFO mapred.Task: Task:attempt_local696442689_0001_r_000000_0 is done. And is in the process of committing
2019-10-13 14:09:30,142 INFO mapred.LocalJobRunner: 3 / 3 copied.
2019-10-13 14:09:30,142 INFO mapred.Task: Task attempt_local696442689_0001_r_000000_0 is allowed to commit now
2019-10-13 14:09:30,143 INFO output.FileOutputCommitter: Saved output of task 'attempt_local696442689_0001_r_000000_0' to file:/home/hadoop/output
2019-10-13 14:09:30,147 INFO mapred.LocalJobRunner: reduce > reduce
2019-10-13 14:09:30,147 INFO mapred.Task: Task 'attempt_local696442689_0001_r_000000_0' done.
2019-10-13 14:09:30,148 INFO mapred.Task: Final Counters for attempt_local696442689_0001_r_000000_0: Counters: 24
File System Counters
FILE: Number of bytes read=317753
FILE: Number of bytes written=816010
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
Map-Reduce Framework
Combine input records=0
Combine output records=0
Reduce input groups=6
Reduce shuffle bytes=130
Reduce input records=9
Reduce output records=6
Spilled Records=9
Shuffled Maps =3
Failed Shuffles=0
Merged Map outputs=3
GC time elapsed (ms)=0
Total committed heap usage (bytes)=168112128
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Output Format Counters
Bytes Written=62
2019-10-13 14:09:30,148 INFO mapred.LocalJobRunner: Finishing task: attempt_local696442689_0001_r_000000_0
2019-10-13 14:09:30,151 INFO mapred.LocalJobRunner: reduce task executor complete.
2019-10-13 14:09:30,574 INFO mapreduce.Job: Job job_local696442689_0001 running in uber mode : false
2019-10-13 14:09:30,575 INFO mapreduce.Job: map 100% reduce 100%
2019-10-13 14:09:30,576 INFO mapreduce.Job: Job job_local696442689_0001 completed successfully
2019-10-13 14:09:30,597 INFO mapreduce.Job: Counters: 30
File System Counters
FILE: Number of bytes read=1269027
FILE: Number of bytes written=3263300
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
Map-Reduce Framework
Map input records=3
Map output records=9
Map output bytes=94
Map output materialized bytes=130
Input split bytes=285
Combine input records=9
Combine output records=9
Reduce input groups=6
Reduce shuffle bytes=130
Reduce input records=9
Reduce output records=6
Spilled Records=18
Shuffled Maps =3
Failed Shuffles=0
Merged Map outputs=3
GC time elapsed (ms)=68
Total committed heap usage (bytes)=654082048
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=55
File Output Format Counters
Bytes Written=62
[hadoop@shengxi ~]$
查看结果：
[hadoop@shengxi home]$ cd hadoop/
[hadoop@shengxi ~]$ ll
total 8
drwxrwxrwx 2 root root 4096 Oct 13 14:00 input
drwxr-xr-x 2 hadoop hadoop 4096 Oct 13 14:09 output
[hadoop@shengxi ~]$ cd output/
[hadoop@shengxi output]$ ll
total 4
-rw-r--r-- 1 hadoop hadoop 50 Oct 13 14:09 part-r-00000
-rw-r--r-- 1 hadoop hadoop 0 Oct 13 14:09 _SUCCESS
[hadoop@shengxi output]$ cat part-r-00000
dfads 1
dfjlaskd 1
hello 3
ldlkjfh 2
my 1
world 1
[hadoop@shengxi output]$
至此，hadoop的单机版就安装好了。下面开始进行伪分布式部署。
7、进行伪分布部署
（1）修改在/usr/local/hadoop/etc/hadoop的各种配置
（2）给hadoop-env.sh，yarn-env.sh，mapred-env.sh增加java_home的连接。我直接将我的sh修改部分cat出来。
[hadoop@shengxi hadoop]$ cat hadoop-env.sh
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.222.b10-1.el7_7.x86_64
#
[hadoop@shengxi hadoop]$ cat mapred-env.sh
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.222.b10-1.el7_7.x86_64
[hadoop@shengxi hadoop]$ cat yarn-env.sh
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.222.b10-1.el7_7.x86_64
（3） 修改core-site.xml。
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<!-- HDFS资源路径 -->
<!--指定namenode的通信地址，默认8020端口 -->
<property>
<name>fs.defaultFS</name>
<value>hdfs://hadoop:9000</value>
</property>
<!-- 在见面界面访问数据使用的用户名 -->
<property>
<name>hadoop.http.staticuser.user</name>
<value>hadoop</value>
</property>
<!-- Hadoop临时文件存放目录 -->
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/hadoop/data/tmp</value>
</property>
<!-- 垃圾文件保留时间(秒) -->
<property>
<name>fs.trash.interval</name>
<value>7200</value>
</property>
</configuration>
（4）创建对应的文件夹
//傻瓜式创建文件夹
[hadoop@shengxi hadoop-3.1.2]$ mkdir data
[hadoop@shengxi hadoop-3.1.2]$ cd data
[hadoop@shengxi data]$ mkdir tmp
[hadoop@shengxi data]$ mkdir namenode
[hadoop@shengxi data]$ mkdir datanode
[hadoop@shengxi data]$ cd ../
[hadoop@shengxi hadoop-3.1.2]$ chmod -R 777 data/
[hadoop@shengxi hadoop-3.1.2]$
（5）修改hdfs-site.xml，副本数常规是3个，但是我们是伪分布，只使用一个就行了。
<configuration>
<!-- 是否进行权限检查 -->
<property>
<name>dfs.permissions.enabled</name>
<value>false</value>
</property>
<!-- 副本数 -->
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<!-- namenode元数据存储路径 -->
<property>
<name>dfs.namenode.name.dir</name>
<value>/usr/local/hadoop/data/namenode</value>
</property>
<!-- 数据存储路径 -->
<property>
<name>dfs.datanode.data.dir</name>
<value>/usr/local/hadoop/data/datanode</value>
</property>
</configuration>（6）修改 mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<!-- mapreduce运行在yarn上面 -->
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>yarn.app.mapreduce.am.env</name>
<value>HADOOP_MAPRED_HOME=/usr/local/hadoop</value>
</property>
<property>
<name>mapreduce.map.env</name>
<value>HADOOP_MAPRED_HOME=/usr/local/hadoop</value>
</property>
<property>
<name>mapreduce.reduce.env</name>
<value>HADOOP_MAPRED_HOME=/usr/local/hadoop</value>
</property>
<property>
<name>mapreduce.map.memory.mb</name>
<value>2048</value>
</property>
<!-- 历史服务器端地址 -->
<property>
<name>mapreduce.jobhistory.address</name>
<value>hadoop:10020</value>
</property>
<!-- 历史服务器web端地址 -->
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>hadoop:19888</value>
</property>
</configuration>
（7）修改yarn-site.xml
<?xml version="1.0"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<configuration>
<!-- resourceManager在哪台机器 hadoop的意义就是虚拟域名 -->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>hadoop</value>
</property>
<!-- 在nodemanager中运行mapreduce服务 -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<!-- 配置日志的聚集功能 -->
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<!-- 设置日志保留时间(7天) -->
<property>
<name>yarn.log-aggregation.retain-seconds</name>
<value>604800</value>
</property>
</configuration>
至此，配置文件修完成，实现格式化hadoop就行了。
hadoop namenode -format[root@shengxi bin]# hadoop namenode -format
WARNING: Use of this script to execute namenode is deprecated.
WARNING: Attempting to execute replacement "hdfs namenode" instead.
2019-10-13 15:21:29,211 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = shengxi/172.17.0.15
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 3.1.2
STARTUP_MSG: classpath = /usr/local/hadoop-3.1.2/etc/hadoop:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/kerby-xdr-1.0.1.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/kerb-server-1.0.1.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/commons-collections-3.2.2.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/commons-net-3.6.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/commons-beanutils-1.9.3.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/jsr305-3.0.0.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/token-provider-1.0.1.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/jersey-json-1.19.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/zookeeper-3.4.13.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/jackson-databind-2.7.8.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/jsr311-api-1.1.1.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/httpclient-4.5.2.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/commons-compress-1.18.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/jul-to-slf4j-1.7.25.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/paranamer-2.3.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/accessors-smart-1.2.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/commons-lang3-3.4.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/jsch-0.1.54.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/jetty-security-9.3.24.v20180605.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/jetty-xml-9.3.24.v20180605.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/jettison-1.1.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/commons-logging-1.1.3.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/kerb-simplekdc-1.0.1.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/kerb-admin-1.0.1.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/kerby-util-1.0.1.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/kerb-identity-1.0.1.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/json-smart-2.3.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/kerb-common-1.0.1.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/snappy-java-1.0.5.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/jetty-webapp-9.3.24.v20180605.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/kerb-client-1.0.1.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/avro-1.7.7.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/log4j-1.2.17.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/jersey-core-1.19.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/nimbus-jose-jwt-4.41.1.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/jackson-core-2.7.8.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/javax.servlet-api-3.1.0.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/jetty-http-9.3.24.v20180605.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/jetty-server-9.3.24.v20180605.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/kerby-asn1-1.0.1.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/jackson-annotations-2.7.8.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/kerb-crypto-1.0.1.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/netty-3.10.5.Final.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/jcip-annotations-1.0-1.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/slf4j-api-1.7.25.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/hadoop-annotations-3.1.2.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/metrics-core-3.2.4.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/jetty-servlet-9.3.24.v20180605.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/commons-io-2.5.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/commons-cli-1.2.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/woodstox-core-5.0.3.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/jaxb-api-2.2.11.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/commons-lang-2.6.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/jersey-server-1.19.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/curator-client-2.13.0.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/kerby-config-1.0.1.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/re2j-1.1.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/asm-5.0.4.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/commons-configuration2-2.1.1.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/commons-math3-3.1.1.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/guava-11.0.2.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/kerby-pkix-1.0.1.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/curator-framework-2.13.0.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/audience-annotations-0.5.0.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/jsp-api-2.1.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/commons-codec-1.11.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/jetty-util-9.3.24.v20180605.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/stax2-api-3.1.4.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/gson-2.2.4.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/jetty-io-9.3.24.v20180605.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/httpcore-4.4.4.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/curator-recipes-2.13.0.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/jersey-servlet-1.19.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/hadoop-auth-3.1.2.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/kerb-util-1.0.1.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/kerb-core-1.0.1.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/lib/htrace-core4-4.1.0-incubating.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/hadoop-kms-3.1.2.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/hadoop-common-3.1.2.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/hadoop-common-3.1.2-tests.jar:/usr/local/hadoop-3.1.2/share/hadoop/common/hadoop-nfs-3.1.2.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/kerby-xdr-1.0.1.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/kerb-server-1.0.1.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/commons-collections-3.2.2.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/okhttp-2.7.5.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/commons-net-3.6.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/commons-beanutils-1.9.3.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/jsr305-3.0.0.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/token-provider-1.0.1.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/jersey-json-1.19.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/zookeeper-3.4.13.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/jackson-databind-2.7.8.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/jsr311-api-1.1.1.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/httpclient-4.5.2.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/jetty-util-ajax-9.3.24.v20180605.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/commons-compress-1.18.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/paranamer-2.3.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/accessors-smart-1.2.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/commons-lang3-3.4.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/jsch-0.1.54.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/jetty-security-9.3.24.v20180605.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/jetty-xml-9.3.24.v20180605.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/jettison-1.1.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/netty-all-4.0.52.Final.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/kerb-simplekdc-1.0.1.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/kerb-admin-1.0.1.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/kerby-util-1.0.1.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/kerb-identity-1.0.1.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/json-smart-2.3.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/kerb-common-1.0.1.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/snappy-java-1.0.5.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/jetty-webapp-9.3.24.v20180605.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/kerb-client-1.0.1.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/avro-1.7.7.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/jersey-core-1.19.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/nimbus-jose-jwt-4.41.1.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/jackson-core-2.7.8.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/jackson-jaxrs-1.9.13.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/javax.servlet-api-3.1.0.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/jetty-http-9.3.24.v20180605.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/jetty-server-9.3.24.v20180605.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/kerby-asn1-1.0.1.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/jackson-annotations-2.7.8.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/kerb-crypto-1.0.1.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/netty-3.10.5.Final.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/jcip-annotations-1.0-1.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/okio-1.6.0.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/hadoop-annotations-3.1.2.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/jackson-xc-1.9.13.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/json-simple-1.1.1.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/jetty-servlet-9.3.24.v20180605.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/commons-io-2.5.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/woodstox-core-5.0.3.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/jaxb-api-2.2.11.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/jersey-server-1.19.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/curator-client-2.13.0.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/kerby-config-1.0.1.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/re2j-1.1.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/asm-5.0.4.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/commons-configuration2-2.1.1.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/commons-math3-3.1.1.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/guava-11.0.2.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/kerby-pkix-1.0.1.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/curator-framework-2.13.0.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/audience-annotations-0.5.0.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/commons-codec-1.11.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/jetty-util-9.3.24.v20180605.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/stax2-api-3.1.4.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/gson-2.2.4.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/jetty-io-9.3.24.v20180605.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/httpcore-4.4.4.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/curator-recipes-2.13.0.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/jersey-servlet-1.19.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/hadoop-auth-3.1.2.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/kerb-util-1.0.1.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/kerb-core-1.0.1.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/lib/htrace-core4-4.1.0-incubating.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/hadoop-hdfs-native-client-3.1.2-tests.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/hadoop-hdfs-client-3.1.2-tests.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/hadoop-hdfs-rbf-3.1.2.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/hadoop-hdfs-3.1.2.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/hadoop-hdfs-native-client-3.1.2.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/hadoop-hdfs-rbf-3.1.2-tests.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/hadoop-hdfs-client-3.1.2.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/hadoop-hdfs-httpfs-3.1.2.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/hadoop-hdfs-nfs-3.1.2.jar:/usr/local/hadoop-3.1.2/share/hadoop/hdfs/hadoop-hdfs-3.1.2-tests.jar:/usr/local/hadoop-3.1.2/share/hadoop/mapreduce/lib/junit-4.11.jar:/usr/local/hadoop-3.1.2/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/usr/local/hadoop-3.1.2/share/hadoop/mapreduce/hadoop-mapreduce-client-app-3.1.2.jar:/usr/local/hadoop-3.1.2/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-3.1.2.jar:/usr/local/hadoop-3.1.2/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.1.2.jar:/usr/local/hadoop-3.1.2/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.1.2-tests.jar:/usr/local/hadoop-3.1.2/share/hadoop/mapreduce/hadoop-mapreduce-client-nativetask-3.1.2.jar:/usr/local/hadoop-3.1.2/share/hadoop/mapreduce/hadoop-mapreduce-client-uploader-3.1.2.jar:/usr/local/hadoop-3.1.2/share/hadoop/mapreduce/hadoop-mapreduce-client-core-3.1.2.jar:/usr/local/hadoop-3.1.2/share/hadoop/mapreduce/hadoop-mapreduce-client-common-3.1.2.jar:/usr/local/hadoop-3.1.2/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-3.1.2.jar:/usr/local/hadoop-3.1.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.2.jar:/usr/local/hadoop-3.1.2/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-3.1.2.jar:/usr/local/hadoop-3.1.2/share/hadoop/yarn:/usr/local/hadoop-3.1.2/share/hadoop/yarn/lib/guice-servlet-4.0.jar:/usr/local/hadoop-3.1.2/share/hadoop/yarn/lib/java-util-1.9.0.jar:/usr/local/hadoop-3.1.2/share/hadoop/yarn/lib/jackson-module-jaxb-annotations-2.7.8.jar:/usr/local/hadoop-3.1.2/share/hadoop/yarn/lib/snakeyaml-1.16.jar:/usr/local/hadoop-3.1.2/share/hadoop/yarn/lib/fst-2.50.jar:/usr/local/hadoop-3.1.2/share/hadoop/yarn/lib/jackson-jaxrs-json-provider-2.7.8.jar:/usr/local/hadoop-3.1.2/share/hadoop/yarn/lib/javax.inject-1.jar:/usr/local/hadoop-3.1.2/share/hadoop/yarn/lib/HikariCP-java7-2.4.12.jar:/usr/local/hadoop-3.1.2/share/hadoop/yarn/lib/guice-4.0.jar:/usr/local/hadoop-3.1.2/share/hadoop/yarn/lib/objenesis-1.0.jar:/usr/local/hadoop-3.1.2/share/hadoop/yarn/lib/jersey-guice-1.19.jar:/usr/local/hadoop-3.1.2/share/hadoop/yarn/lib/metrics-core-3.2.4.jar:/usr/local/hadoop-3.1.2/share/hadoop/yarn/lib/mssql-jdbc-6.2.1.jre7.jar:/usr/local/hadoop-3.1.2/share/hadoop/yarn/lib/aopalliance-1.0.jar:/usr/local/hadoop-3.1.2/share/hadoop/yarn/lib/jersey-client-1.19.jar:/usr/local/hadoop-3.1.2/share/hadoop/yarn/lib/jackson-jaxrs-base-2.7.8.jar:/usr/local/hadoop-3.1.2/share/hadoop/yarn/lib/dnsjava-2.1.7.jar:/usr/local/hadoop-3.1.2/share/hadoop/yarn/lib/swagger-annotations-1.5.4.jar:/usr/local/hadoop-3.1.2/share/hadoop/yarn/lib/json-io-2.5.1.jar:/usr/local/hadoop-3.1.2/share/hadoop/yarn/lib/ehcache-3.3.1.jar:/usr/local/hadoop-3.1.2/share/hadoop/yarn/lib/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:/usr/local/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-registry-3.1.2.jar:/usr/local/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-3.1.2.jar:/usr/local/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-3.1.2.jar:/usr/local/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-server-tests-3.1.2.jar:/usr/local/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-server-timeline-pluginstorage-3.1.2.jar:/usr/local/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.2.jar:/usr/local/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-client-3.1.2.jar:/usr/local/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-server-nodemanager-3.1.2.jar:/usr/local/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-services-api-3.1.2.jar:/usr/local/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-server-web-proxy-3.1.2.jar:/usr/local/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-server-common-3.1.2.jar:/usr/local/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-services-core-3.1.2.jar:/usr/local/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-server-router-3.1.2.jar:/usr/local/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-3.1.2.jar:/usr/local/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-common-3.1.2.jar:/usr/local/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-api-3.1.2.jar:/usr/local/hadoop-3.1.2/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-3.1.2.jar
STARTUP_MSG: build = https://github.com/apache/hadoop.git -r 1019dde65bcf12e05ef48ac71e84550d589e5d9a; compiled by 'sunilg' on 2019-01-29T01:39Z
STARTUP_MSG: java = 1.8.0_222
************************************************************/
2019-10-13 15:21:29,232 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
2019-10-13 15:21:29,421 INFO namenode.NameNode: createNameNode [-format]
2019-10-13 15:21:30,291 INFO common.Util: Assuming 'file' scheme for path /usr/local/hadoop/data/namenode in configuration.
2019-10-13 15:21:30,291 INFO common.Util: Assuming 'file' scheme for path /usr/local/hadoop/data/namenode in configuration.
Formatting using clusterid: CID-d1d9f073-058a-4ff6-9edb-abf48551e43c
2019-10-13 15:21:30,345 INFO namenode.FSEditLog: Edit logging is async:true
2019-10-13 15:21:30,362 INFO namenode.FSNamesystem: KeyProvider: null
2019-10-13 15:21:30,363 INFO namenode.FSNamesystem: fsLock is fair: true
2019-10-13 15:21:30,365 INFO namenode.FSNamesystem: Detailed lock hold time metrics enabled: false
2019-10-13 15:21:30,373 INFO namenode.FSNamesystem: fsOwner = root (auth:SIMPLE)
2019-10-13 15:21:30,374 INFO namenode.FSNamesystem: supergroup = supergroup
2019-10-13 15:21:30,374 INFO namenode.FSNamesystem: isPermissionEnabled = false
2019-10-13 15:21:30,374 INFO namenode.FSNamesystem: HA Enabled: false
2019-10-13 15:21:30,434 INFO common.Util: dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling
2019-10-13 15:21:30,448 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit: configured=1000, counted=60, effected=1000
2019-10-13 15:21:30,448 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
2019-10-13 15:21:30,454 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
2019-10-13 15:21:30,454 INFO blockmanagement.BlockManager: The block deletion will start around 2019 Oct 13 15:21:30
2019-10-13 15:21:30,456 INFO util.GSet: Computing capacity for map BlocksMap
2019-10-13 15:21:30,458 INFO util.GSet: VM type = 64-bit
2019-10-13 15:21:30,459 INFO util.GSet: 2.0% max memory 444.7 MB = 8.9 MB
2019-10-13 15:21:30,459 INFO util.GSet: capacity = 2^20 = 1048576 entries
2019-10-13 15:21:30,469 INFO blockmanagement.BlockManager: dfs.block.access.token.enable = false
2019-10-13 15:21:30,483 INFO Configuration.deprecation: No unit for dfs.namenode.safemode.extension(30000) assuming MILLISECONDS
2019-10-13 15:21:30,483 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
2019-10-13 15:21:30,483 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.min.datanodes = 0
2019-10-13 15:21:30,483 INFO blockmanagement.BlockManagerSafeMode: dfs.namenode.safemode.extension = 30000
2019-10-13 15:21:30,483 INFO blockmanagement.BlockManager: defaultReplication = 1
2019-10-13 15:21:30,483 INFO blockmanagement.BlockManager: maxReplication = 512
2019-10-13 15:21:30,483 INFO blockmanagement.BlockManager: minReplication = 1
2019-10-13 15:21:30,483 INFO blockmanagement.BlockManager: maxReplicationStreams = 2
2019-10-13 15:21:30,484 INFO blockmanagement.BlockManager: redundancyRecheckInterval = 3000ms
2019-10-13 15:21:30,484 INFO blockmanagement.BlockManager: encryptDataTransfer = false
2019-10-13 15:21:30,484 INFO blockmanagement.BlockManager: maxNumBlocksToLog = 1000
2019-10-13 15:21:30,538 INFO namenode.FSDirectory: GLOBAL serial map: bits=24 maxEntries=16777215
2019-10-13 15:21:30,552 INFO util.GSet: Computing capacity for map INodeMap
2019-10-13 15:21:30,552 INFO util.GSet: VM type = 64-bit
2019-10-13 15:21:30,552 INFO util.GSet: 1.0% max memory 444.7 MB = 4.4 MB
2019-10-13 15:21:30,552 INFO util.GSet: capacity = 2^19 = 524288 entries
2019-10-13 15:21:30,565 INFO namenode.FSDirectory: ACLs enabled? false
2019-10-13 15:21:30,566 INFO namenode.FSDirectory: POSIX ACL inheritance enabled? true
2019-10-13 15:21:30,566 INFO namenode.FSDirectory: XAttrs enabled? true
2019-10-13 15:21:30,566 INFO namenode.NameNode: Caching file names occurring more than 10 times
2019-10-13 15:21:30,571 INFO snapshot.SnapshotManager: Loaded config captureOpenFiles: false, skipCaptureAccessTimeOnlyChange: false, snapshotDiffAllowSnapRootDescendant: true, maxSnapshotLimit: 65536
2019-10-13 15:21:30,573 INFO snapshot.SnapshotManager: SkipList is disabled
2019-10-13 15:21:30,580 INFO util.GSet: Computing capacity for map cachedBlocks
2019-10-13 15:21:30,580 INFO util.GSet: VM type = 64-bit
2019-10-13 15:21:30,580 INFO util.GSet: 0.25% max memory 444.7 MB = 1.1 MB
2019-10-13 15:21:30,580 INFO util.GSet: capacity = 2^17 = 131072 entries
2019-10-13 15:21:30,587 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10
2019-10-13 15:21:30,587 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10
2019-10-13 15:21:30,587 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25
2019-10-13 15:21:30,596 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
2019-10-13 15:21:30,596 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
2019-10-13 15:21:30,598 INFO util.GSet: Computing capacity for map NameNodeRetryCache
2019-10-13 15:21:30,598 INFO util.GSet: VM type = 64-bit
2019-10-13 15:21:30,598 INFO util.GSet: 0.029999999329447746% max memory 444.7 MB = 136.6 KB
2019-10-13 15:21:30,598 INFO util.GSet: capacity = 2^14 = 16384 entries
2019-10-13 15:21:30,640 INFO namenode.FSImage: Allocated new BlockPoolId: BP-1558567234-172.17.0.15-1570951290630
2019-10-13 15:21:30,682 INFO common.Storage: Storage directory /usr/local/hadoop/data/namenode has been successfully formatted.
2019-10-13 15:21:30,690 INFO namenode.FSImageFormatProtobuf: Saving image file /usr/local/hadoop/data/namenode/current/fsimage.ckpt_0000000000000000000 using no compression
2019-10-13 15:21:30,792 INFO namenode.FSImageFormatProtobuf: Image file /usr/local/hadoop/data/namenode/current/fsimage.ckpt_0000000000000000000 of size 391 bytes saved in 0 seconds .
2019-10-13 15:21:30,813 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
2019-10-13 15:21:30,819 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at shengxi/172.17.0.15
************************************************************/
[root@shengxi bin]#
最重要的是下面这一段中的successfully formatted.和txid>=0有些版本还会返回status=0。
2019-10-13 15:21:30,682 INFO common.Storage: Storage directory /usr/local/hadoop/data/namenode has been successfully formatted.
2019-10-13 15:21:30,690 INFO namenode.FSImageFormatProtobuf: Saving image file /usr/local/hadoop/data/namenode/current/fsimage.ckpt_0000000000000000000 using no compression
2019-10-13 15:21:30,792 INFO namenode.FSImageFormatProtobuf: Image file /usr/local/hadoop/data/namenode/current/fsimage.ckpt_0000000000000000000 of size 391 bytes saved in 0 seconds .
2019-10-13 15:21:30,813 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
（8）启动环境，注意2.x和3.x是不一样的。
* 2.x的是
sbin/hadoop-daemon.sh start namenode
sbin/hadoop-daemon.sh start datanode
sbin/yarn-daemon.sh start resourcemanager
sbin/yarn-daemon.sh start nodemanager
sbin/mr-jobhistory-daemon.sh start historyserver
* 3.x的启动方式是：
hdfs --daemon start namenode
hdfs --daemon start datanode
yarn --daemon start resourcemanager
yarn --daemon start nodemanager
yarn --daemon start timelineserver
结果如下：
[root@shengxi sbin]# hdfs --daemon start namenode
[root@shengxi sbin]# hdfs --daemon start datanode
[root@shengxi sbin]# yarn --daemon start resourcemanager
WARNING: YARN_CONF_DIR has been replaced by HADOOP_CONF_DIR. Using value of YARN_CONF_DIR.
[root@shengxi sbin]# yarn --daemon start nodemanager
WARNING: YARN_CONF_DIR has been replaced by HADOOP_CONF_DIR. Using value of YARN_CONF_DIR.
[root@shengxi sbin]# yarn --daemon start timelineserver
WARNING: YARN_CONF_DIR has been replaced by HADOOP_CONF_DIR. Using value of YARN_CONF_DIR验证：
[root@shengxi hadoop-3.1.2]# jps
721 DataNode
610 NameNode
1268 ApplicationHistoryServer
1111 NodeManager
844 ResourceManager
1293 Jps
[root@shengxi hadoop-3.1.2]#
进行webUI检测（注意：云控制台要开启对应的端口）
web管理 |2.x端口| 3.x端口
NameNode | 8020 | 9820
NameNode HTTP UI | 50070 | 9870
DataNode | 50010 | 9866
Secondary NameNode HTTP UI | 50090 | 9868
DataNode IPC | 50020 | 9867
DataNode HTTP UI | 50075 | 9864
DataNode | 50010 | 9866
将单机测试的input文件夹copy到分布式里面，真实路径就是在hdfs-site.xml设定的。即：/usr/local/hadoop/data/datanode
利用web新建文件夹input将home里面的input文件夹中的文件上传到input中。
[root@shengxi ~]# hdfs dfs -put input/* /input
[root@shengxi ~]#
开始测试，命令是
hadoop jar /usr/local/hadoop-3.1.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.2.jar wordcount /input /output
运行过程如下
[root@shengxi ~]# hadoop jar /usr/local/hadoop-3.1.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.2.jar wordcount /input /output
2019-10-13 15:49:46,920 INFO client.RMProxy: Connecting to ResourceManager at hadoop/172.17.0.15:8032
2019-10-13 15:49:47,690 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1570951999899_0002
2019-10-13 15:49:48,063 INFO input.FileInputFormat: Total input files to process : 3
2019-10-13 15:49:48,961 INFO mapreduce.JobSubmitter: number of splits:3
2019-10-13 15:49:49,708 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1570951999899_0002
2019-10-13 15:49:49,710 INFO mapreduce.JobSubmitter: Executing with tokens: []
2019-10-13 15:49:49,967 INFO conf.Configuration: resource-types.xml not found
2019-10-13 15:49:49,968 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2019-10-13 15:49:50,048 INFO impl.YarnClientImpl: Submitted application application_1570951999899_0002
2019-10-13 15:49:50,109 INFO mapreduce.Job: The url to track the job: http://hadoop:8088/proxy/application_1570951999899_0002/
2019-10-13 15:49:50,110 INFO mapreduce.Job: Running job: job_1570951999899_0002
2019-10-13 15:49:58,477 INFO mapreduce.Job: Job job_1570951999899_0002 running in uber mode : false
2019-10-13 15:49:58,479 INFO mapreduce.Job: map 0% reduce 0%
2019-10-13 15:50:13,719 INFO mapreduce.Job: map 100% reduce 0%
2019-10-13 15:50:21,799 INFO mapreduce.Job: map 100% reduce 100%
2019-10-13 15:50:23,828 INFO mapreduce.Job: Job job_1570951999899_0002 completed successfully
2019-10-13 15:50:23,930 INFO mapreduce.Job: Counters: 53
File System Counters
FILE: Number of bytes read=118
FILE: Number of bytes written=864161
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=343
HDFS: Number of bytes written=50
HDFS: Number of read operations=14
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=3
Launched reduce tasks=1
Data-local map tasks=3
Total time spent by all maps in occupied slots (ms)=75502
Total time spent by all reduces in occupied slots (ms)=5829
Total time spent by all map tasks (ms)=37751
Total time spent by all reduce tasks (ms)=5829
Total vcore-milliseconds taken by all map tasks=37751
Total vcore-milliseconds taken by all reduce tasks=5829
Total megabyte-milliseconds taken by all map tasks=77314048
Total megabyte-milliseconds taken by all reduce tasks=5968896
Map-Reduce Framework
Map input records=3
Map output records=9
Map output bytes=94
Map output materialized bytes=130
Input split bytes=288
Combine input records=9
Combine output records=9
Reduce input groups=6
Reduce shuffle bytes=130
Reduce input records=9
Reduce output records=6
Spilled Records=18
Shuffled Maps =3
Failed Shuffles=0
Merged Map outputs=3
GC time elapsed (ms)=832
CPU time spent (ms)=1980
Physical memory (bytes) snapshot=716304384
Virtual memory (bytes) snapshot=13735567360
Total committed heap usage (bytes)=436482048
Peak Map Physical memory (bytes)=205828096
Peak Map Virtual memory (bytes)=3649929216
Peak Reduce Physical memory (bytes)=106946560
Peak Reduce Virtual memory (bytes)=2791301120
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=55
File Output Format Counters
Bytes Written=50
[root@shengxi ~]#
在命令行中查看
hdfs dfs -cat /output/*//结果为
[root@shengxi ~]# hdfs dfs -cat /output/*
dfads 1
dfjlaskd 1
hello 3
ldlkjfh 2
my 1
world 1
[root@shengxi ~]#
结果是和单机版一模一样的。
在web中也可以看到结果就是成功了。