手工安装CDH Hadoop 单结点

官方文档

https://www.cloudera.com/documentation/enterprise/latest/topics/cdh_ig_cdh5_install.html#concept_gp2_q32_24
http://www.cloudera.com/documentation/cdh/5-0-x/CDH5-Quick-Start/cdh5qs_mrv1_pseudo.html

先决条件

安装好JDK, 设置好JAVA_HOME

设置cloudera的repo

1. cd /etc/yum.repos.d/
2. sudo wget https://archive.cloudera.com/cdh5/redhat/7/x86_64/cdh/cloudera-cdh5.repo
3. sudo rpm –import https://archive.cloudera.com/cdh5/redhat/7/x86_64/cdh/RPM-GPG-KEY-cloudera

重点

sudo yum install hadoop-conf-pseudo-2.6.0+cdh5.12.0+2512-1.cdh5.12.0.p0.38.el7.x86_64.rpm

format namenode

sudo -u hdfs hdfs namenode -format

start datanode

sudo -u hdfs hdfs namenode

start services

for i incd /etc/init.d;ls hadoop-*; do sudo service $i start; done

sudo -u hdfs hadoop fs -mkdir -p /tmp
sudo -u hdfs hadoop fs -chmod -R 1777 /tmp
sudo -u hdfs hadoop fs -mkdir -p /var/lib/hadoop-hdfs/cache/mapred/mapred/staging
sudo -u hdfs hadoop fs -chmod 1777 /var/lib/hadoop-hdfs/cache/mapred/mapred/staging
sudo -u hdfs hadoop fs -chown -R mapred /var/lib/hadoop-hdfs/cache/mapred

其它

发现在腾讯的clould server上用yum 安装超级慢, 所以要手工下载了所有的rpm包, 然后上传到server上安装
手工安装时如果有包的依赖无法安装, 可以用rpm -ivh name.rpm –nodeps –force强制安装

Incompatible clusterIDs 错误是因为重复format namenode, 把datanode上的data 目录全部删除, data目录是在dfs.datanode.data.dir指定的

yarn.web-proxy.address is not set so the proxy will not run, 增加

<property>
<name>yarn.web-proxy.address</name>
<value>localhost:8100></value>
</property>

到yarn-site.xml

安装组件

sudo yum install hadoop-yarn-resourcemanager
sudo yum install hadoop-hdfs-namenode
sudo yum install hadoop-hdfs-secondarynamenode
sudo yum install hadoop-yarn-nodemanager hadoop-hdfs-datanode hadoop-mapreduce
sudo yum install hadoop-mapreduce-historyserver hadoop-yarn-proxyserver
sudo yum install hadoop-client