Hadoop2.6.x安装详细教程

Hadoop是基于java实现的,因此必须安装java环境。以下为实现hadoop2.0安装教程。

1.在主节点的 /usr/local/src/下进行对java解压

tar xvzf jdk-7u67-linux-x64.tar.gz

配置java环境变量

#Master、Slave1、Slave2

vim ~/.bashrc

export JAVA_HOME=/usr/local/src/jdk1.7.0_67

export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib

export PATH=$PATH:$JAVA_HOME/bin

刷新环境变量

source ~/.bashrc

远程拷贝到slave1和slave2(192.168.**.**为从节点的ip地址)

scp-rp jdk1.7.0_67 192.168.**.**:/usr/local/src/

scp-rp jdk1.7.0_67 192.168.**.**:/usr/local/src/

 2.在主节点对hadoop压缩包进行解压

命令行输入:tar xvzf hadoop-2.6.1.tar.gz

3.进入:/usr/local/src/hadoop-2.6.1/etc/hadoop目录

a)hadoop-env.sh配置JAVA_HOME

export JAVA_HOME=/usr/local/src/jdk1.7.0_67

 b)yarn-env.sh配置JAVA_HOME

exportJAVA_HOME=/usr/local/src/jdk1.7.0_67

c)slaves配置从节点

vim slaves

slave1

slave2

d)配置core-site.xml

vimcore-site.xml

<property>

         <name>fs.defaultFS</name>

        <value>hdfs://master:9000/</value>

     </property>

     <property>

        <name>hadoop.tmp.dir</name>

        <value>file:/usr/local/src/hadoop-2.6.1/tmp</value>

     </property> 

e)配置hdfs-site.xml

vimhdfs-site.xml

<property>

       <name>dfs.namenode.secondary.http-address</name>

        <value>master:9001</value>

     </property>

     <property>

        <name>dfs.namenode.name.dir</name>

         <value>file:/usr/local/src/hadoop-2.6.1/dfs/name</value>

     </property>

     <property>

        <name>dfs.datanode.data.dir</name>

        <value>file:/usr/local/src/hadoop-2.6.1/dfs/data</value>

     </property>

     <property>

         <name>dfs.replication</name>

         <value>3</value>

     </property>

f)配置mapred-site.xml

cpmapred-site.xml.template mapred-site.xml

vimmapred-site.xml

  <property>

       <name>mapreduce.framework.name</name>

        <value>yarn</value>

</property> 

g)配置yarn-site.xml

vimyarn-site.xml

  <property>

        <name>yarn.nodemanager.aux-services</name>

        <value>mapreduce_shuffle</value>

    </property>

    <property>

       <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

        <value>org.apache.hadoop.mapred.ShuffleHandler</value>

    </property>

    <property>

       <name>yarn.resourcemanager.address</name>

        <value>master:8032</value>

    </property>

    <property>

       <name>yarn.resourcemanager.scheduler.address</name>

       <value>master:8030</value>

    </property>

    <property>

       <name>yarn.resourcemanager.resource-tracker.address</name>

        <value>master:8035</value>

    </property>

    <property>

       <name>yarn.resourcemanager.admin.address</name>

        <value>master:8033</value>

    </property>

    <property>

       <name>yarn.resourcemanager.webapp.address</name>

        <value>master:8088</value>

</property> 

h)在HAOOP_HOME(usr/local/src)创建目录:

mkdirtmp

mkdir-p dfs/name

mkdir-p dfs/data 

3.配置环境变量

#Master、Slave1、Slave2

vim ~/.bashrc

export HADOOP_HOME =/usr/local/src/hadoop-2.6.1

export PATH=$PATH:$HADOOP_HOME/bin

#刷新环境变量

source ~/.bashrc

4. 本地网络配置

#Master、Slave1、Slave2

vim/etc/hosts

在hosts中把主从节点的ip地址写进去

192.168.**.** master

192.168.**.** slave1

192.168.**.** slave2

host没有生效,需要生效(临时生效)

在命令行输入:hostname master (#Master)

                       hostname slave1 (#Slave1)

                       hostname slave2 (#Slave2)

改为永久生效:

vim/etc/sysconfig/network

HOSTNAME=master(#Master)

HOSTNAME=slave1(#Slave1)

HOSTNAME=slave2(#Slave1)

5.拷贝安装包

#Master

scp-rp hadoop-2.6.1 192.168.**.**:/usr/local/src/

scp-rp hadoop-2.6.1 192.168.**.**:/usr/local/src/

6.关闭防火墙

#Master、Slave1、Slave2

/ect/init.d/iptablesstop

chkconfig--level 35 iptables off (永久关闭)

检查是否关闭成功:iptables –L

输入:setenforce 0

输入: getenforce 查看是否关闭了

7.建立每台机器的互信关系

a)#Master、Slave1、Slave2

ssh-keygen然后回车然后回车(回两次车)

进入隐藏目录:cd ~/.ssh/ 然后ls

id_rsa.pub是共钥文件,id_rsa是密钥文件

id_rsa.pub这个公钥文件的内容拷贝给authorized_keys

b)#Slave1、#Slave2

ssh-keygen然后回车然后回车(回两次车)

cd~/.ssh/ 然后ls

cat id_rsa.pub

c)将slave1、slave2的公钥id_rsa.pub复制到主机节点authorized_keys

vim authorized_keys

d)将主节点上的authorized_keys分别分发给slave1和slave2两个节点

命令行输入:scp -r pauthorized_keys slave1:~/.ssh/

命令行输入:scp -r pauthorized_keys slave2:~/.ssh/

 e)验证:分别在主节点和2个从节点上验证,如:

在主节点上命令行上输入:ssh slave1退出:exit

在主机节点上登录:ssh slave2

 8.启动集群

#Master

#初始化Namenode

cd/usr/local/src/hadoop-2.6.1/bin

./hadoop namenode –format

#启动集群

cd/usr/local/src/hadoop-2.6.1

./sbin/start-all.sh

9.集群状态

#Master、Slave1、Slave2

jps查看主节点和从节点的进程。


关于为什么要配置公钥?
公钥配置完之后各节点的登录不需要登录密码,因为tasktracker每隔3秒会向jobtracker发出询问(hadoop1.0),若没有公钥会频繁的输入密码,造成繁琐(在hadoop2.0中tasktracker叫做NodeManager,jobtracker包括ResourceManager和ApplicationMaster)。


猜你喜欢

转载自blog.csdn.net/qq_38332574/article/details/80339022