Hadoop2.7.2安装

版权声明:本文为博主原创文章,未经博主允许不得转载。 https://blog.csdn.net/Bactryki28/article/details/68062809

安装前准备

首先需要安装JDK并配置java环境变量,我安装的是jdk1.8.1。
下载hadoop-2.7.2.tar.gz,解压到:/usr/hadoop,
解压之后进行配置就可以使用

配置hadoop

环境变量配置

export JAVA_HOME=/usr/java/jdk1.8.0_101
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export HADOOP_HOME=/usr/hadoop/hadoop-2.7.2
export PATH=$PATH:$HADOOP_HOME/bin
export PATH=$PATH:$HADOOP_HOME/sbin
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"
export PATH=${JAVA_HOME}/bin:$PATH

hadoop的环境变量还是要配的,否则无法进行namenode的格式化操作。
设置了环境变量需要刷新用source命令重新解析:

source ~/.hashrc

配置$HADOOP_HOME/etc/core-site.xml

<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://localhost:9000</value>
    </property>
    <!-- 指定hadoop运行时产生文件的存储路径 -->
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/opt/soft/hadoop-2.7.2/tmp</value>
    </property>
</configuration>

配置$HADOOP_HOME/etc/hdfs-site.xml

<configuration>
    <property>  
        <name>dfs.replication</name>  
        <value>1</value>  
    </property>  
</configuration>

配置$HADOOP_HOME/etc/mapred-site.xml

<configuration>
    <property>  
        <name>mapreduce.framework.name</name>  
        <value>yarn</value>  
    </property>  
</configuration>

不知道是不是我下载的版本的问题,没有mapred-site.xml这个文件,是新建一个加进去的。而etc目录下有个mapred-site.xml.template文件,也一起配置相同的内容。这里不去追究它是怎么来的干嘛用,先把环境搭起来。

配置$HADOOP_HOME/etc/yarn-site.xml

<configuration>

<!-- Site specific YARN configuration properties -->
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
</configuration>

配置$HADOOP_HOME/bin/hadoop-env.sh

export JAVA_HOME=${JAVA_HOME}

改为

export JAVA_HOME=/usr/java/jdk1.8.0_101

将:

export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop"}

改为

export HADOOP_CONF_DIR=/usr/hadoop/hadoop-2.7.2/etc/hadoop

否则在启动hadoop执行start-all.sh的时候会报:

localhost: Error: JAVA_HOME is not set and could not be found.
localhost: Error: JAVA_HOME is not set and could not be found.

格式化namenode

执行:

lzw@ubuntu:/usr/hadoop/hadoop-2.7.2/bin$ hadoop namenode -format

中间没报异常就是执行成功了。

设置ssh免密码登录

依次执行以下三个命令

ssh-keygen -t rsa

ssh-keygen代表生成秘钥;
-t(区分大小写)生成的秘钥类型;
rsa秘钥认证(即秘药类型);
这个命令将为用户lzw生成其密钥对,询问其保存路径时直接回车采用默认路径,当提示要为生成的密钥输入passphrase的时候,直接回车,也就是将其设定为空密码。生成的密钥对id_rsa,id_rsa.pub,默认存储在/home/lzw/.ssh目录下然后将id_rsa.pub的内容复制到每个机器(也包括本机)的/home/dbrg/.ssh/authorized_keys文件中,如果机器上已经有authorized_keys这个文件了,就在文件末尾加上id_rsa.pub中的内容,如果没有authorized_keys这个文件,直接复制过去就行)

进入.ssh目录:

cd .ssh

执行:

cp id_rsa.pub  authorized_keys

测试:

lzw@ubuntu:~/.ssh$ ssh master
Welcome to Ubuntu 14.04.4 LTS (GNU/Linux 4.2.0-27-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

0 packages can be updated.
0 updates are security updates.

New release '16.04.2 LTS' available.
Run 'do-release-upgrade' to upgrade to it.

Last login: Fri Mar 31 14:22:24 2017 from master

第一次连接会提示输入yes,之后就不需要了

复制authorized_keys到slave1

scp authorized_keys slave1:/home/lzw/.ssh/

附录:重启ssh服务

service ssh restart

验证是否成功

命令行进入/usr/hadoop/hadoop-2.7.2/sbin
执行:start-all.sh
执行完毕:

This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
17/03/29 15:03:18 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [localhost]
localhost: starting namenode, logging to /usr/hadoop/hadoop-2.7.2/logs/hadoop-root-namenode-ubuntu.out
localhost: starting datanode, logging to /usr/hadoop/hadoop-2.7.2/logs/hadoop-root-datanode-ubuntu.out
Starting secondary namenodes [0.0.0.0]
The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.
ECDSA key fingerprint is b4:8c:ff:91:54:a0:89:4e:3f:7a:88:76:1f:c8:18:5a.
Are you sure you want to continue connecting (yes/no)? yes
0.0.0.0: Warning: Permanently added '0.0.0.0' (ECDSA) to the list of known hosts.
0.0.0.0: starting secondarynamenode, logging to /usr/hadoop/hadoop-2.7.2/logs/hadoop-root-secondarynamenode-ubuntu.out
17/03/29 15:03:38 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
starting yarn daemons
starting resourcemanager, logging to /usr/hadoop/hadoop-2.7.2/logs/yarn-root-resourcemanager-ubuntu.out
localhost: starting nodemanager, logging to /usr/hadoop/hadoop-2.7.2/logs/yarn-root-nodemanager-ubuntu.out

执行jps:

root@ubuntu:/usr/hadoop/hadoop-2.7.2/sbin# jps
8372 NodeManager
9173 Jps
8087 SecondaryNameNode
7912 DataNode
3901 ResourceManager

出现如上进程说明安装成功了。

猜你喜欢

转载自blog.csdn.net/Bactryki28/article/details/68062809