本次hadoop源码包版本信息：hadoop-2.8.1-src.tar.gz，如有需要请到http://hadoop.apache.org这个网址去下载。
PS：编译是一般企业会用到，学习的小伙伴就不需要编译，可直接下载官网编译好的文件部署。就是不带src的文件包。不需要编译的小伙伴可以直接跳到Hadoop部署哦。
Hadoop
- 广义:
以Hadoop为主的生态圈
- 狭义:
Apache Hadoop2.X 3.X组件
HDFS 分布式存储MapReduce 分布式计算 job1
Yarn 资源和作业调度和管理平台 Spark
Apache Hadoop1.x组件
HDFS 分布式存储
MapReduce 分布式计算 + 资源和作业调度和管理
相关网站
http://hadoop.apache.org/ http://hive.apache.org/ http://spark.apache.org/ http://github.com/apache/hadoop
Hadoop编译
hadoop编译目的：1、打补丁（例如：系统bug，需要更改jar包后，编译后重新部署到环境中）；2、定制化
xxx-srcxxx.tar.gz 源代码包 比较小的 不带src的 编译之后的包 大
1、创建源码和编译文件夹
cd /opt mkdir software sourcecode
2、在sourcecode目录下上传软件包并解压
rz tar -xzvf hadoop-2.8.1-src.tar.gz
3、查看hadoop-2.8.1-src文件夹中的BULIDING.txt文件夹，查看安装需求
组件的安装请参考小编另外一篇博文：Hadoop编译环境组件搭建
4、编译，查看BUILDING文档，选择Create binary distribution with native code and with documentation
mvn package -Pdist,native,docs -DskipTests -Dtar
执行以上命令会说没有POM文件，故要在hadoop解压的文件夹下面去执行该命令
PS：该编译命令需要执行30分钟左右，已经提供了.m2.gz的包，但是还是要联网校验，没有的话，可能需要3.5h
编译好的tar包保存在
/opt/sourcecode/hadoop-2.8.1-src/hadoop-dist/target/hadoop-2.8.1.tar.gz
PS：
- 有时候编译过程中会出现下载某个包的时间太久，这是由于连接网站的过程中会出现假死，此时按ctrl+c，重新运行编译命令。
- 如果出现缺少了某个文件的情况，则要先清理maven(使用命令 mvn clean) 再重新编译。
main:
[exec] $ tar cf hadoop-2.8.1.tar hadoop-2.8.1
[exec] $ gzip -f hadoop-2.8.1.tar
[exec]
[exec] Hadoop dist tar available at: /opt/sourcecode/hadoop-2.8.1-src/hadoop-dist/target/hadoop-2.8.1.tar.gz
[exec]
[INFO] Executed tasks
[INFO]
[INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ hadoop-dist ---
[INFO] Building jar: /opt/sourcecode/hadoop-2.8.1-src/hadoop-dist/target/hadoop-dist-2.8.1-javadoc.jar
[INFO]
[INFO] --- maven-source-plugin:2.3:jar-no-fork (hadoop-java-sources) @ hadoop-dist ---
[INFO] Building jar: /opt/sourcecode/hadoop-2.8.1-src/hadoop-dist/target/hadoop-dist-2.8.1-sources.jar
[INFO]
[INFO] --- maven-source-plugin:2.3:test-jar-no-fork (hadoop-java-sources) @ hadoop-dist ---
[INFO] Building jar: /opt/sourcecode/hadoop-2.8.1-src/hadoop-dist/target/hadoop-dist-2.8.1-test-sources.jar
[INFO]
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (dist-enforce) @ hadoop-dist ---
[INFO]
[INFO] --- maven-site-plugin:3.5:attach-descriptor (attach-descriptor) @ hadoop-dist ---
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [ 2.102 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 1.658 s]
。。。
[INFO] Apache Hadoop Distribution ......................... SUCCESS [01:09 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 18:38 min
[INFO] Finished at: 2018-06-01T22:14:32+08:00
[INFO] Final Memory: 164M/452M
[INFO] ------------------------------------------------------------------------
如上所示，hadoop编译完成，可以开始部署
Hadoop部署
hadoop文件夹说明
- bin：存放对hadoop相关服务（HDFS，YARN）进行操作的脚本
- sbin：存放启动或停止hadoop相关服务的脚本
- etc：hadoop的配置文件目录，存放hadoop的配置文件
- share：存放hadoop的依赖jar包和文档，文档可以被删除掉
- lib：存放hadoop的本地库（对数据进行压缩解压缩功能）
准备工作
1、创建hadoop用户，并passwd hadoop修改密码
id hadoop #没有 useradd hadoop
2、hadoop部署方式
http://hadoop.apache.org/docs/r2.8.4/hadoop-project-dist/hadoop-common/SingleCluster.html Standalone Operation 单机模式 不开发 不生产 没有进程 Pseudo-Distributed Operation 伪分布式 开发 不生产 独立进程 但是只部署在1台机器上 Fully-Distributed Operation 完全分布式 开发 生产 独立进程 但是只部署在N（>=3）台机器上
3、检查/etc/hosts文件夹
cat /etc/hosts
如上已经配置好IP和机器名对应关系，可通过一下命令测试
ping hadoop01
成功
4、解压编译好的tar包，并ll查看
tar -xzvf hadoop-2.8.1.tar.gz
5、更改文件夹用户和用户组权限
chown -R hadoop:hadoop hadoop-2.8.1
HDFS伪分布式部署
1、切换到hadoop用户，进入到如下目录，配置文件
vi etc/hadoop/core-site.xml
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://192.168.233.111:9000</value>
</property>
</configuration>
ps：core-site.xml文件其他内容不动，在<configuration>中插入内容，伪分布式地址填写自己主机IP
继续修改文件
vi etc/hadoop/hdfs-site.xml
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
2、Setup passphraseless ssh
$ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa $ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys $ chmod 0600 ~/.ssh/authorized_keys配置完成后，测试如下表示成功
3、Format the filesystem:
bin/hdfs namenode -format
[hadoop@hadoop01 hadoop-2.8.1]$ bin/hdfs namenode -format
18/06/01 23:19:27 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: user = hadoop
STARTUP_MSG: host = hadoop01/192.168.233.111
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 2.8.1
STARTUP_MSG: classpath = /opt/software/hadoop-2.8.1/etc/hadoop:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/hamcrest-core-1.3.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/slf4j-api-1.7.10.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/json-smart-1.1.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/commons-codec-1.4.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/curator-client-2.7.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/htrace-core4-4.0.1-incubating.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/commons-logging-1.1.3.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/hadoop-auth-2.8.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/jersey-core-1.9.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/nimbus-jose-jwt-3.9.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/gson-2.2.4.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/commons-io-2.4.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/log4j-1.2.17.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/commons-compress-1.4.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/jetty-sslengine-6.1.26.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/jsch-0.1.51.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/xz-1.0.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/curator-framework-2.7.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/httpclient-4.5.2.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/mockito-all-1.8.5.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/servlet-api-2.5.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/httpcore-4.4.4.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/junit-4.11.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/jetty-util-6.1.26.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/commons-net-3.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/guava-11.0.2.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/netty-3.6.2.Final.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/jsr305-3.0.0.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/commons-collections-3.2.2.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/paranamer-2.3.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/asm-3.2.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/jetty-6.1.26.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/jersey-server-1.9.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/xmlenc-0.52.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/jsp-api-2.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/commons-digester-1.8.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/commons-math3-3.1.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/curator-recipes-2.7.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/jettison-1.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/commons-lang-2.6.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/commons-configuration-1.6.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/commons-cli-1.2.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/zookeeper-3.4.6.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/jcip-annotations-1.0.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/avro-1.7.4.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/hadoop-annotations-2.8.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/stax-api-1.0-2.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/jersey-json-1.9.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/activation-1.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/lib/jets3t-0.9.0.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/hadoop-common-2.8.1-tests.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/hadoop-nfs-2.8.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/common/hadoop-common-2.8.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/hdfs:/opt/software/hadoop-2.8.1/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/opt/software/hadoop-2.8.1/share/hadoop/hdfs/lib/htrace-core4-4.0.1-incubating.jar:/opt/software/hadoop-2.8.1/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/opt/software/hadoop-2.8.1/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/opt/software/hadoop-2.8.1/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/opt/software/hadoop-2.8.1/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/opt/software/hadoop-2.8.1/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar:/opt/software/hadoop-2.8.1/share/hadoop/hdfs/lib/commons-io-2.4.jar:/opt/software/hadoop-2.8.1/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/opt/software/hadoop-2.8.1/share/hadoop/hdfs/lib/hadoop-hdfs-client-2.8.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/opt/software/hadoop-2.8.1/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/opt/software/hadoop-2.8.1/share/hadoop/hdfs/lib/guava-11.0.2.jar:/opt/software/hadoop-2.8.1/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/opt/software/hadoop-2.8.1/share/hadoop/hdfs/lib/jsr305-3.0.0.jar:/opt/software/hadoop-2.8.1/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/opt/software/hadoop-2.8.1/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/opt/software/hadoop-2.8.1/share/hadoop/hdfs/lib/asm-3.2.jar:/opt/software/hadoop-2.8.1/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/opt/software/hadoop-2.8.1/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/opt/software/hadoop-2.8.1/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/opt/software/hadoop-2.8.1/share/hadoop/hdfs/lib/okhttp-2.4.0.jar:/opt/software/hadoop-2.8.1/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/opt/software/hadoop-2.8.1/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/opt/software/hadoop-2.8.1/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/opt/software/hadoop-2.8.1/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/opt/software/hadoop-2.8.1/share/hadoop/hdfs/lib/okio-1.4.0.jar:/opt/software/hadoop-2.8.1/share/hadoop/hdfs/hadoop-hdfs-nfs-2.8.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/hdfs/hadoop-hdfs-2.8.1-tests.jar:/opt/software/hadoop-2.8.1/share/hadoop/hdfs/hadoop-hdfs-client-2.8.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/hdfs/hadoop-hdfs-client-2.8.1-tests.jar:/opt/software/hadoop-2.8.1/share/hadoop/hdfs/hadoop-hdfs-native-client-2.8.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/hdfs/hadoop-hdfs-native-client-2.8.1-tests.jar:/opt/software/hadoop-2.8.1/share/hadoop/hdfs/hadoop-hdfs-2.8.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/lib/javassist-3.18.1-GA.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/lib/fst-2.24.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/lib/commons-codec-1.4.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/lib/javax.inject-1.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/lib/curator-client-2.7.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/lib/aopalliance-1.0.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/lib/jersey-core-1.9.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/lib/commons-io-2.4.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/lib/commons-math-2.2.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/lib/jersey-client-1.9.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/lib/log4j-1.2.17.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/lib/curator-test-2.7.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/lib/xz-1.0.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/lib/servlet-api-2.5.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/lib/guava-11.0.2.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/lib/jsr305-3.0.0.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/lib/commons-collections-3.2.2.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/lib/objenesis-2.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/lib/guice-3.0.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/lib/asm-3.2.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/lib/jetty-6.1.26.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/lib/jersey-server-1.9.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/lib/jettison-1.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/lib/commons-lang-2.6.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/lib/commons-cli-1.2.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/lib/zookeeper-3.4.6-tests.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/lib/jersey-json-1.9.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/lib/activation-1.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/hadoop-yarn-server-common-2.8.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.8.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/hadoop-yarn-api-2.8.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.8.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/hadoop-yarn-client-2.8.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.8.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/hadoop-yarn-server-tests-2.8.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.8.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.8.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/hadoop-yarn-common-2.8.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/hadoop-yarn-registry-2.8.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.8.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/hadoop-yarn-server-timeline-pluginstorage-2.8.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.8.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/opt/software/hadoop-2.8.1/share/hadoop/mapreduce/lib/javax.inject-1.jar:/opt/software/hadoop-2.8.1/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/opt/software/hadoop-2.8.1/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/opt/software/hadoop-2.8.1/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/opt/software/hadoop-2.8.1/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/opt/software/hadoop-2.8.1/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/opt/software/hadoop-2.8.1/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/opt/software/hadoop-2.8.1/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/mapreduce/lib/xz-1.0.jar:/opt/software/hadoop-2.8.1/share/hadoop/mapreduce/lib/junit-4.11.jar:/opt/software/hadoop-2.8.1/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/opt/software/hadoop-2.8.1/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:/opt/software/hadoop-2.8.1/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/opt/software/hadoop-2.8.1/share/hadoop/mapreduce/lib/guice-3.0.jar:/opt/software/hadoop-2.8.1/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/opt/software/hadoop-2.8.1/share/hadoop/mapreduce/lib/asm-3.2.jar:/opt/software/hadoop-2.8.1/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/opt/software/hadoop-2.8.1/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/opt/software/hadoop-2.8.1/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/opt/software/hadoop-2.8.1/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/opt/software/hadoop-2.8.1/share/hadoop/mapreduce/lib/hadoop-annotations-2.8.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.8.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.8.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.8.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.8.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.8.1-tests.jar:/opt/software/hadoop-2.8.1/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.8.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.8.1.jar:/opt/software/hadoop-2.8.1/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.8.1.jar:/contrib/capacity-scheduler/*.jar
STARTUP_MSG: build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r 20fe5304904fc2f5a18053c389e43cd26f7a70fe; compiled by 'vinodkv' on 2017-06-02T06:14Z
STARTUP_MSG: java = 1.8.0_45
************************************************************/
18/06/01 23:19:31 INFO common.Storage: Storage directory /tmp/hadoop-hadoop/dfs/name has been successfully formatted.
18/06/01 23:19:31 INFO namenode.FSImageFormatProtobuf: Saving image file /tmp/hadoop-hadoop/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
18/06/01 23:19:31 INFO namenode.FSImageFormatProtobuf: Image file /tmp/hadoop-hadoop/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 322 bytes saved in 0 seconds.
18/06/01 23:19:31 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
18/06/01 23:19:31 INFO util.ExitUtil: Exiting with status 0
18/06/01 23:19:31 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at hadoop01/192.168.233.111
************************************************************
格式化成功
Start NameNode daemon and DataNode daemon:
4、启动HDFS
sbin/start-dfs.sh
出现错误，配置即可
vi etc/hadoop/hadoop-env.sh export JAVA_HOME=/usr/java/jdk1.8.0_45
重新启动
[hadoop@hadoop01 hadoop-2.8.1]$ sbin/start-dfs.sh Starting namenodes on [hadoop01] hadoop01: starting namenode, logging to /opt/software/hadoop-2.8.1/logs/hadoop-hadoop-namenode-hadoop01.out localhost: starting datanode, logging to /opt/software/hadoop-2.8.1/logs/hadoop-hadoop-datanode-hadoop01.out Starting secondary namenodes [0.0.0.0] 0.0.0.0: starting secondarynamenode, logging to /opt/software/hadoop-2.8.1/logs/hadoop-hadoop-secondarynamenode-hadoop01.out
PS：关于HDFS的启动，由显示内容可知，namenode进程的启动IP为设置好得主机ip：hadoop01，datanode为localhost，secondarynamenode为0.0.0.0，关于启动ip的更改，详见小编另外一篇博文：HDFS三进程启动IP更改教程
[hadoop@hadoop01 hadoop-2.8.1]$ jps 18736 NameNode 19012 SecondaryNameNode 19133 Jps 18831 DataNode
PS：jps详细教程见小编另外一篇博文：jps详解
HDFS伪分布式三个进程简称 NameNode NN DataNode DN SecondaryNameNode SNN
5、浏览Namenode的Web窗口，默认方式为：
NameNode - http://localhost:50070/ localhost为主机IP
登录成功，HDFS部署成功（web服务需要端口号时，可用命令：netstat -nlp | grep pid 查询端口号，在用web登录查看，若服务启动，端口号正确，登录不上，可以考虑防火墙问题）
PS：HDFS启动和关闭可以设置用户环境变量哦，参考如下，别忘了设置完source哦
cd ~ vi .bash_profile #HDFS Start and Stop alias start-dfs='/opt/software/hadoop-2.8.1/sbin/start-dfs.sh' alias stop-dfs='/opt/software/hadoop-2.8.1/sbin/stop-dfs.sh'
HDFS补充；
创建执行MapReduce作业所需的HDFS目录
$ bin/hdfs dfs -mkdir /user $ bin/hdfs dfs -mkdir /user/<username>
我们在查看了linux目录，在根目录下并没有/user这个文件夹，这里需要解释一下。
linux系统 存储 部署服务 / linux系统有存储功能，部署的服务是从根目录/开始查询
hdfs 分布式文件系统 存储 / hdfs相当于架构在linux系统上的分布式文件存储系统，也是从根目录/开始查询
创建好文件夹以后，这时我们就可以在web界面上查看到了。如下图所示。
这里面有一个细节，那就是在/前面省略了一个东西，我们进入配置文件：/opt/software/hadoop-2.8.1/etc/hadoop，cat一下core-site.xml文件。fs.defaultFS默认的路径地址就是下面框选的东西，也就是文件的入口就是ip+端口号。完整的就是：
bin/hdfs dfs -mkdir hdfs://192.168.233.111:9000/user
YARN配置
MapReduce的job运行在YARN上，故我们先进行YARN配置
1、切换到配置文件目录
cd /opt/software/hadoop-2.8.1/etc/hadoop
2、复制配置文件，配置mapred-site.xml文件
cp mapred-site.xml.template mapred-site.xml vi mapred-site.xml配置内容如下
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
3、配置yarn-site.xml文件
vi yarn-site.xml
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>
4、启动yarn
sbin/start-yarn.sh
启动成功。多了两个进程：ResourceManager（老大，简写RM）和 NodeManager （小弟，简写NM）
5、查看yarn的ResourceManager的web窗口
http://localhost:8088/
就此YARN的伪分布式部署已经完成，MapReduce不需要部署，它是运行在YARN上的。