Hadoop-2.6.0-cdh5.7.0 HDFS部署

1、添加sudo权限的无密码访问的hadoop用户

[root@hadoop002 ~]# useradd hadoop
[root@hadoop002 ~]# cat /etc/sudoers |grep hadoop
hadoop  ALL=(ALL)       NOPASSWD: ALL
[root@hadoop002 ~]# 
[root@hadoop002 ~]# su - hadoop
[hadoop@hadoop002 ~]$ 

2、下载并安装

[hadoop@hadoop002 ~]$ mkdir app 
[hadoop@hadoop002 ~]$ cd app
[hadoop@hadoop002 app]$ wget http://archive.cloudera.com/cdh5/cdh/5/hadoop-2.6.0-cdh5.7.0.tar.gz
[hadoop@hadoop002 app]$ tar -xzvf hadoop-2.6.0-cdh5.7.0.tar.gz
[hadoop@hadoop002 app]$ cd hadoop-2.6.0-cdh5.7.0
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ 

3、配置文件

[hadoop@hadoop002 hadoop]$ vi core-site.xml 
<configuration>
    <property>
        <name>fs.defaultFS</name>
        <value>hdfs://hadoop002:9000</value>
    </property>
</configuration>
"core-site.xml" 24L, 884C written       

[hadoop@hadoop002 hadoop]$ vi hdfs-site.xml 
<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration>
"hdfs-site.xml" 23L, 866C written         

6、配置环境变量

[hadoop@hadoop002 ~]$ vi .bash_profile 
export MVN_HOME=/home/hadoop/app/apache-maven-3.3.9
export PROTOC_HOME=/home/hadoop/app/protobuf
export FINDBUGS_HOME=/home/hadoop/app/findbugs-1.3.9
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs
export JAVA_HOME=/usr/java/jdk1.7.0_80
export HADOOP_PREFIX=/home/hadoop/app/hadoop-2.6.0-cdh5.7.0

export PATH=$HADOOP_PREFIX/bin:$JAVA_HOME/bin:$PATH
~
~
".bash_profile" 12L, 293C written                         

在集群环境下，即使各节点都正确地配置了JAVA_HOME，也会报JAVA_HOME is note set and could not be found.

解决方法： 在/etc/hadoop/hadoop-env.sh中，再显示地重新声明一遍JAVA_HOME,如下所示:

export JAVA_HOME=/home/hadoop/app/jdk1.7.0_80

5、创建ssh免密登

[hadoop@hadoop002 hadoop]$ cd
[hadoop@hadoop002 ~]$ 
[hadoop@hadoop002 ~]$ 
[hadoop@hadoop002 ~]$ rm -rf .ssh
[hadoop@hadoop002 ~]$ ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
Generating public/private dsa key pair.
Created directory '/home/hadoop/.ssh'.
Your identification has been saved in /home/hadoop/.ssh/id_dsa.
Your public key has been saved in /home/hadoop/.ssh/id_dsa.pub.
The key fingerprint is:
a3:c7:ba:e9:2e:77:ff:6f:50:bd:bc:f7:1b:1d:a6:e1 hadoop@hadoop002
The key's randomart image is:
+--[ DSA 1024]----+
|                 |
|                 |
|               . |
|              . .|
|        S    o.o.|
|       o .  o +oo|
|      . o    E .o|
|    . .+.     ..o|
|     =*o ....o..=|
+-----------------+
[hadoop@hadoop002 ~]$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
[hadoop@hadoop002 ~]$ cd .ssh
[hadoop@hadoop002 .ssh]$ ll
total 12
-rw-rw-r-- 1 hadoop hadoop 606 Sep 19 23:16 authorized_keys
-rw------- 1 hadoop hadoop 668 Sep 19 23:16 id_dsa
-rw-r--r-- 1 hadoop hadoop 606 Sep 19 23:16 id_dsa.pub

[hadoop@hadoop002 .ssh]$ chmod 600 authorized_keys
[hadoop@hadoop002 .ssh]$ 

[hadoop@hadoop002 .ssh]$ ssh hadoop002
The authenticity of host 'hadoop002 (192.168.191.140)' can't be established.
RSA key fingerprint is b1:94:33:ec:95:89:bf:06:3b:ef:30:2f:d7:8e:d2:4c.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'hadoop002,192.168.191.140' (RSA) to the list of known hosts.
Last login: Wed Sep 19 18:21:09 2018 from 192.168.191.140

6、格式化名称节点并启动HDFS

[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ bin/hdfs namenode -format
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ sbin/start-dfs.sh
[hadoop@hadoop002 /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/etc/hadoop]$jps
6149 DataNode
6577 Jps
6062 NameNode
6337 SecondaryNameNode

本机访问http://192.168.191.140:50070

至此成功启动并运行HDFS

如果发现启动时DataNode进程有问题，需重新部署

[root@hadoop002 tmp]# rm -rf /tmp/hadoop-hadoop
[root@hadoop002 tmp]# 
[hadoop@hadoop002 hadoop]$ vi slaves 
hadoop002
[hadoop@hadoop002 hadoop]$ cd ../../
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ 
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ bin/hdfs namenode -format
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ sbin/start-dfs.sh
18/09/19 23:29:20 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [hadoop002]
hadoop002: starting namenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-namenode-hadoop002.out
hadoop002: starting datanode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-datanode-hadoop002.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/logs/hadoop-hadoop-secondarynamenode-hadoop002.out
18/09/19 23:29:36 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ 
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ 
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$ jps
28288 NameNode
28686 Jps
28410 DataNode
28575 SecondaryNameNode
[hadoop@hadoop002 hadoop-2.6.0-cdh5.7.0]$