hadoop分布式集群安装

hadoop分布式集群安装

1、配置ssh无密码登录
安装ssh后
将slaves的ip加入/etc/hosts中
执行cd ~ ，输入ssh-keygen后连续3次回车
cat .ssh/id_rsa.pub >> .ssh/authorized_keys
分发主机公钥
scp .ssh/id_rsa.pub root@slave:/root/master.pub
cat master.pub >> .ssh/authorized_keys

测试ssh slave

#ssh master
ssh-keygen -t rsa -p ''
cat ~/.ssh/id_rsa.pub >>  ~/.ssh/authorized_keys

for slave in $HADOOP_HOME/conf/slaves) do
 scp ~/.ssh/id_rsa.pub root@slave:/root/master.pub
done

#for slaves
domain=""
master="192.168.0.100 $1 $domain"
slave="192.168.0.$3 $2 $domain"

ssh-keygen -t rsa -p ''
cat ~/.ssh/id_rsa.pub >>  ~/.ssh/authorized_keys
cat /root/master.pub >> ~/.ssh/authorized_keys

echo $master >> /etc/hosts
echo $slave >> /etc/hosts

mkdir /home/hadoop

#for master
for slave in $HADOOP_HOME/conf/slaves) do
 scp $HADOOP_HOME root@$slave:$HADOOP_HOME
done

#sync conf
for slave in $HADOOP_HOME/conf/slaves) do
 scp $HADOOP_HOME/conf/*-site.xml root@$slave:$HADOOP_HOME/conf/
 scp $HADOOP_HOME/conf/masters root@$slave:$HADOOP_HOME/conf/
 scp $HADOOP_HOME/conf/slaves root@$slave:$HADOOP_HOME/conf/
done

2、安装java1.6
到java官方网站下载解压版jdk1.6或以上
tar -xzf jdk-*-tar.gz

3、安装hadoop
下载hadoop-*-tar.gz 并解压

3、配置hadoop
vi /etc/profile
追加以下hadoop环境变量及hadoop快捷命令
export JAVA_HOME=/usr/jdk1.7.0_03
export HADOOP_HOME=/home/hadoop/hadoop-0.20.203
export PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$PATH

alias hls='hadoop fs -ls'
alias hcat='hadoop fs -cat'
alias hput='hadoop fs -put'
alias hget='hadoop fs -get'
alias hrmr='hadoop fs -rmr'
alias hmkdir='hadoop fs -mkdir'

alias hjar='hadoop jar'

修改bin/hadoop加入export JAVA_HOME=/usr/jdk1.7.0_03后分发

修改site.xml后分发

配置masters、slaves文件。

4、启动集群
start-all.sh
jps查看启动情况

查看master机器上的http服务是否与预期一致。

http://192.168.0.100:50030/查看jobtracker的情况，http://192.168.0.100:50070/查看dfs情况

5、测试集群

在master上执行以下命令测试测试dfs及mr。
hls /
hmkdir input
hput conf/*.xml input
通过hjar来执行wordcount来测试map reduce

hjar hadoop-example*.jar wordcount