一.安装hadoop
(这里需要有一定的docker知识)
1.安装镜像
我们抛弃了传统的vm方式,使用docker来安装部署hadoop.首先要准备一个镜像，可以使用Dockerfile构建一个合适自己的镜像，或者可以在共有仓库中找一个具有hadoop环境的镜像来使用也可以。由于我是配置的阿里云的加速器，所以在阿里云的仓库中找了一个具有hadoop环境的镜像。hadoop镜像地址
使用命令拉到本地
docker pull registry.cn-hangzhou.aliyuncs.com/kaibb/hadoop
下载完成之后，通过docker images 可以查看到该镜像：
registry.cn-hangzhou.aliyuncs.com/kaibb/hadoop latest 44c31aee79de 24 months ago 927MB
2.创建容器
有了镜像之后，我们根据该镜像创建三个容器，分别是一个Master用来作为hadoop集群的namenode，剩下两个Slave用来作为datanode。
可以使用命令：
docker run -i -t --name Master -h Master registry.cn-hangzhou.aliyuncs.com/kaibb/hadoop /bin/bash
命令中的-h为该容器设置了主机名，这里设置为Master，否则创建容器之后是一串随机的数字和字母不好辨认使用。–name指定了容器的名字，也是为了方便区分使用。
建立好Master节点的容器之后，再分别使用两次该命令，创建两个Slave节点。稍微改变其中的参数即可：
例如创建Slave1节点：
docker run -i -t --name Slave1 -h Slave1 registry.cn-hangzhou.aliyuncs.com/kaibb/hadoop /bin/bash
这样集群的基本环境就准备好了。
3.配置ssh (为了多台机器无密码互相访问)
首先在master将SSH运行起来。
/etc/init.d/ssh start
然后生成秘钥，保存到authorized_keys中。
root@Master:/# ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa): /root/.ssh/id_rsa already exists. Overwrite (y/n)? y Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: 47:43:2d:03:fa:93:ce:00:8a:3f:4d:bb:67:09:65:df root@Master The key's randomart image is: +---[RSA 2048]----+ | .... | | . .o . | | . . oo | | . . .o. o . | |. . .o..S.. | | . o.. +.oE | | o o. .o | | . .+ | | .o | +-----------------+ root@Master:/# cd root@Master:~# cd .ssh/ root@Master:~/.ssh# cat id_rsa id_rsa id_rsa.pub root@Master:~/.ssh# cat id_rsa id_rsa id_rsa.pub root@Master:~/.ssh# cat id_rsa.pub > authorized_keys root@Master:~/.ssh# cat authorized_keys ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQ2SQCcb1Jxf4hpVlbWkCpywmR4r0zeAuzQ6AUBFDHqfXgMQj/ku5s0xtbjjQWlKShvAiV1wLSwuAHfzKfKG0RFhcAXJDTBrDQnN9lBKeEcD1XlBHwtm8N+37bQWSXDLimx1UsNWkUy3rPrB55kV8ofTztI+sZMoOcS34Scib3hjmrbAMC3O0naJkBSaTch7KyX9q3kzxmMixShgBdxPAp6qLIzluIxtI/GXV7M9lfY1+KOWfD01jzHH2sAWpeIrefmEtLwNlv4YRuk0uHRaXOmnfyNj2K4WhTEhJpYdK9KJ3jNXiZZNGKZt3UGDSowPMHfmmUC+Jr3imr3YyP9VFh root@Master
接着在两个Slave上也执行同样的操作
接着使用ip addr查看ip
root@Master:~/.ssh# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1
link/ipip 0.0.0.0 brd 0.0.0.0
3: ip6tnl0@NONE: <NOARP> mtu 1452 qdisc noop state DOWN group default qlen 1
link/tunnel6 :: brd ::
6: eth0@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
valid_lft forever preferred_lft forever
然后在/etc/hosts里面把所有机器的ip都配置好
172.17.0.2 Master 172.17.0.3 Slave1 172.17.0.4 Slave2
把master的公钥copy到从节点
.
ssh-copy-id -i Slave2 ssh-copy-id -i Slave1
接着执行ssh Slave1就可以成功了,exit退出ssh
4.配置hadoop
hadoop-env.sh：修改有关java的环境(镜像里面已经帮我们配置好了)
export JAVA_HOME=/opt/tools/jdk1.8.0_77
core-site.xml(定位文件系统的nameNode)
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://Master:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/hadoop/tmp</value>
</property>
</configuration>
mapred-site.xml(定位JobTraker所在主节点)
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>Master:9001</value>
</property>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
hdfs-site.xml(hdfs配置)
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/hadoop/data</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/hadoop/name</value>
</property>
</configuration>
yarn-site.xml
<configuration>
<property>
<name>yarn.resourcemanager.address</name>
<value>Master:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>Master:8030</value> </property> <property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>Master:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>Master:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>Master:8088</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>
slaves文件
把各个节点都加进去
Slave1 Slave2
然后将这些文件通过scp发送到各个Slave节点上覆盖原来的文件：
scp core-site.xml hadoop-env.sh hdfs-site.xml mapred-site.xml yarn-site.xml Slave1:/opt/tools/hadoop/etc/hadoop/
5.运行hadoop
首先要格式化hadoop
hadoop namenode -format
然后启动hadoop并用jps查看进程
。
root@Master:/opt/tools/hadoop/sbin# start-all.sh This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh Starting namenodes on [Master] Master: starting namenode, logging to /opt/tools/hadoop-2.7.2/logs/hadoop-root-namenode-Master.out Slave2: starting datanode, logging to /opt/tools/hadoop-2.7.2/logs/hadoop-root-datanode-Slave2.out Slave1: starting datanode, logging to /opt/tools/hadoop-2.7.2/logs/hadoop-root-datanode-Slave1.out Starting secondary namenodes [0.0.0.0] 0.0.0.0: starting secondarynamenode, logging to /opt/tools/hadoop-2.7.2/logs/hadoop-root-secondarynamenode-Master.out starting yarn daemons starting resourcemanager, logging to /opt/tools/hadoop/logs/yarn--resourcemanager-Master.out Slave2: starting nodemanager, logging to /opt/tools/hadoop-2.7.2/logs/yarn-root-nodemanager-Slave2.out Slave1: starting nodemanager, logging to /opt/tools/hadoop-2.7.2/logs/yarn-root-nodemanager-Slave1.out root@Master:/opt/tools/hadoop/sbin# jps 2097 ResourceManager 1747 NameNode 1941 SecondaryNameNode 873 NodeManager 2346 Jps 463 DataNode
使用命令 hadoop dfsadmin -report 查看各个节点的状态
DEPRECATED: Use of this script to execute hdfs command is deprecated. Instead use the hdfs command for it. Configured Capacity: 188176871424 (175.25 GB) Present Capacity: 168504856576 (156.93 GB) DFS Remaining: 168504778752 (156.93 GB) DFS Used: 77824 (76 KB) DFS Used%: 0.00% Under replicated blocks: 0 Blocks with corrupt replicas: 0 Missing blocks: 0 Missing blocks (with replication factor 1): 0 ------------------------------------------------- Live datanodes (3): Name: 172.17.0.2:50010 (Master) Hostname: Master Decommission Status : Normal Configured Capacity: 62725623808 (58.42 GB) DFS Used: 28672 (28 KB) Non DFS Used: 6557335552 (6.11 GB) DFS Remaining: 56168259584 (52.31 GB) DFS Used%: 0.00% DFS Remaining%: 89.55% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 1 Last contact: Sat Apr 14 14:35:46 UTC 2018 Name: 172.17.0.4:50010 (Slave2) Hostname: Slave2 Decommission Status : Normal Configured Capacity: 62725623808 (58.42 GB) DFS Used: 24576 (24 KB) Non DFS Used: 6557339648 (6.11 GB) DFS Remaining: 56168259584 (52.31 GB) DFS Used%: 0.00% DFS Remaining%: 89.55% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 1 Last contact: Sat Apr 14 14:35:48 UTC 2018 Name: 172.17.0.3:50010 (Slave1) Hostname: Slave1 Decommission Status : Normal Configured Capacity: 62725623808 (58.42 GB) DFS Used: 24576 (24 KB) Non DFS Used: 6557339648 (6.11 GB) DFS Remaining: 56168259584 (52.31 GB) DFS Used%: 0.00% DFS Remaining%: 89.55% Configured Cache Capacity: 0 (0 B) Cache Used: 0 (0 B) Cache Remaining: 0 (0 B) Cache Used%: 100.00% Cache Remaining%: 0.00% Xceivers: 1 Last contact: Sat Apr 14 14:35:48 UTC 2018