hadoop3.1.3完全分布模式安装

准备：三台centos7

	   jdk1.8
	   hadoop3.1.3

三台分别设置静态ip为

master:192.168.152.100
node1:192.168.152.101
node2:192.168.152.102

修改网卡（这是master的）

vi /etc/sysconfig/network-scripts/ifcfg-ens32

TYPE=Ethernet
BOOTPROTO=static
NAME=ens32
DEVICE=ens32
ONBOOT=yes
IPADDR=192.168.152.100
GATEWAY=192.168.152.2
NETMASK=255.255.255.0
DNS1=8.8.8.8

systemctl restart network
systemctl stop firewalld
setenforce 0

配置免密登录

vi /etc/hosts

192.168.152.100 master
192.168.152.101 node1
192.168.152.102 node2

cp给另外两台node

scp /etc/hosts node1:/etc/hosts
scp /etc/hosts node2:/etc/hosts
ssh-keygen -t rsa(三台都要)
cd /root/.ssh/
cat id_rsa.pub>>authorized_keys
将authorized_keys拷贝的node1
然后将id_rsa.pub加到authorized_keys中在拷贝给node2
然后将id_rsa.pub加到authorized_keys中再拷贝给另外两台
此时的authorized_keys为获取三台的keys

配置jdk

将jdk压缩包放到/opt下并解压
vi /etc/profile

#java environment
export JAVA_HOME=/opt/jdk1.8.0_231
export CLASSPATH=.:${JAVA_HOME}/jre/lib/rt.jar:${JAVA_HOME}/lib/dt.jar:${JAVA_HOME}/lib/tools.jar
export PATH=$PATH:${JAVA_HOME}/bin

source /etc/profile
java -version
出现java版本即安装成功

配置hadoop

将hadoop安装包放到/opt下并解压
vi /etc/profile

#hadoop environment
export HADOOP_HOME=/opt/hadoop-3.1.3
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin

source /etc/profile

mkdir /usr/local/hadoop
mkdir /usr/local/hadoop/data
mkdir /usr/local/hadoop/data/tmp
mkdir /usr/local/hadoop/dfs
mkdir /usr/local/hadoop/dfs/data
mkdir /usr/local/hadoop/dfs/name
mkdir /usr/local/hadoop/tmp

cd /opt/hadoop-3.1.3/etc/hadoop/
vi hadoop-env.sh

export JAVA_HOME=/opt/jdk1.8.0_231

vi core-site.xml

 <configuration>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/usr/local/hadoop/tmp</value>
    </property>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://master:9000</value>
    </property>
</configuration>

vi hdfs-site.xml

<configuration>
    <property>
        <name>dfs.namenode.http-address</name>
        <value>master:50070</value>
    </property>
    <property><!--namenode持久存储名字空间及事务日志的本地文件系统路径-->
        <name>dfs.namenode.name.dir</name>
        <value>/usr/local/hadoop/dfs/name</value>
     </property>
     <property><!--DataNode存放块数据的本地文件系统路径-->
         <name>dfs.datanode.data.dir</name>
         <value>/usr/local/hadoop/dfs/data</value>
     </property>
     <property><!--数据需要备份的数量，不能大于集群的机器数量，默认为3-->
        <name>dfs.replication</name>
        <value>2</value>
    </property>
</configuration>

vi yarn-site.xml

<configuration>
    <property><!--NodeManager上运行的附属服务，用于运行mapreduce-->
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.address</name>
        <value>master:8088</value>
    </property>
</configuration>

vi mapred-site.xml

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>

vi workers
删除localhost改成

node1
node2

vi /opt/hadoop-3.1.3/sbin/start-yarn.sh
vi /opt/hadoop-3.1.3/sbin/stop-yarn.sh
加入

YARN_RESOURCEMANAGER_USER=root
HADOOP_SECURE_DN_USER=yarn
YARN_NODEMANAGER_USER=root

vi /opt/hadoop-3.1.3/sbin/start-dfs.sh
vi /opt/hadoop-3.1.3/sbin/stop-dfs.sh

HDFS_DATANODE_USER=root
HDFS_DATANODE_SECURE_USER=hdfs
HDFS_NAMENODE_USER=root
HDFS_SECONDARYNAMENODE_USER=root

将改的文件全部分发给子节点

scp -r /opt/hadoop-3.1.3/etc/hadoop/ node1:/opt/hadoop-3.1.3/etc/hadoop/
scp -r /opt/hadoop-3.1.3/etc/hadoop/ node2:/opt/hadoop-3.1.3/etc/hadoop/
scp -r /opt/hadoop-3.1.3/sbin/ node1:/opt/hadoop-3.1.3/sbin/
scp -r /opt/hadoop-3.1.3/sbin/ node2:/opt/hadoop-3.1.3/sbin/
scp /opt/hadoop-3.1.3/etc/hadoop/* node1:/opt/hadoop-3.1.3/etc/hadoop/
scp /opt/hadoop-3.1.3/etc/hadoop/* node2:/opt/hadoop-3.1.3/etc/hadoop/

hadoop classpath
复制路径
然后vi /opt/hadoop-3.1.3/etc/hadoop/yarn-site.xml
在configuration中加入(value值是刚复制的路径)

<property>
        <name>yarn.resourcemanager.hostname</name>
        <value>master</value>
</property>
<property>
   <name>yarn.application.classpath</name>
<value>/opt/hadoop-3.1.3/etc/hadoop:/opt/hadoop-3.1.3/share/hadoop/common/lib/*:/opt/hadoop-3.1.3/share/hadoop/common/*:/opt/hadoop-3.1.3/share/hadoop/hdfs:/opt/hadoop-3.1.3/share/hadoop/hdfs/lib/*:/opt/hadoop-3.1.3/share/hadoop/hdfs/*:/opt/hadoop-3.1.3/share/hadoop/mapreduce/lib/*:/opt/hadoop-3.1.3/share/hadoop/mapreduce/*:/opt/hadoop-3.1.3/share/hadoop/yarn:/opt/hadoop-3.1.3/share/hadoop/yarn/lib/*:/opt/hadoop-3.1.3/share/hadoop/yarn/*</value>
</property>

分发
scp /opt/hadoop-3.1.3/etc/hadoop/* node1:/opt/hadoop-3.1.3/etc/hadoop/
scp /opt/hadoop-3.1.3/etc/hadoop/* node2:/opt/hadoop-3.1.3/etc/hadoop/

初始化hadoop

cd /opt/hadoop-3.1.3/bin
./hadoop namenode -format

启动hadoop

cd /opt/hadoop-3.1.3/sbin
./start-all.sh
访问http://192.168.152.100:8088
http://192.168.152.100:50070

hadoop跑自带的wordcount程序

hdfs dfs -ls /
hdfs dfs -mkdir /input
向hdfs上传一个文件作为跑wordcount的文本
hdfs dfs -put /etc/httpd/conf/httpd.conf /input
cd /opt/hadoop-3.1.3/share/hadoop/mapreduce/
hadoop jar hadoop-mapreduce-examples-3.1.3.jar wordcount /input/httpd.conf /output
hdfs dfs -cat /output/part-r-00000
这里注意自己的版本

如果hadoop运行报错进入安全模式
执行命令退出安全模式

hadoop dfsadmin -safemode leave

执行健康检查，删除损坏掉的block

hdfs fsck  /  -delete