CentOS6安装搭建Cloudera Manager 5和CDH5(版本5.13.3)大数据集群

系统环境: 
四台虚拟主机 
操作系统:CentOS 6(8核,16G,512G硬盘) 
Cloudera Manager:5.13.3 
CDH: 5.13.3

192.168.1.xx bigdata1 
192.168.1.xx bigdata2 
192.168.1.xx bigdata3 
192.168.1.xx bigdata4

相关资源下载: 
1.Cloudera Manager13.3下载地址: 
http://archive.cloudera.com/cm5/cm/5/cloudera-manager-el6-cm5.13.3_x86_64.tar.gz 
其他版本自行寻找: 
http://archive.cloudera.com/cm5/cm/5/ 
2.CDH安装包地址: 
http://archive.cloudera.com/cdh5/parcels/5.13.3.2/

同样对应操作系统下载:

CDH-5.13.3-1.cdh5.13.3.p0.2-el6.parcel 
CDH-5.13.3-1.cdh5.13.3.p0.2-el6.parcel.sha1 
manifest.json

一、系统环境搭建(详细步骤略)

大概就是: 
1.没改好静态IP的设置主机IP

vi /etc/sysconfig/network-scripts/ifcfg-eth0

里面大概长这样:
DEVICE=eth0
TYPE=Ethernet
UUID=1c81710f-f99e-4f6e-a3fa-7c8bb20b0xxx
ONBOOT=yes
NM_CONTROLLED=yes
BOOTPROTO=none
HWADDR=00:0C:29:28:C5:52
IPADDR=192.168.x.xxx
PREFIX=24
GATEWAY=192.168.x.x
DEFROUTE=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
NAME="System eth0"

2.重启网卡

service network restart

3.修改主机名

vi /etc/sysconfig/network

4.关闭防火墙

chkconfig iptables off
service iptables stop

5.关闭SELinux(centOS需要)

vi /etc/selinux/config
#修改SELINUX=disabled

6.配置 /etc/hosts 把主机都填进去

vi /etc/hosts

192.168.1.xxx bigdata1
192.168.1.xxx bigdata2
192.168.1.xxx bigdata3
192.168.1.xxx bigdata4

7.重启机器 
8.打通SSH免密码登录: 
生成密钥:ssh-keygen -t rsa 
复制秘钥到每台机器:ssh-copy-id -i <用户名>@<主机名> 
8.安装配置JDK1.8+ 
解压到/usr/目录下,配置profile

二、主节点配置

1.在主节点安装配置mysql

2.登录mysql数据库 
mysql -uroot -p密码

3.创建所需库

--创建hive数据库
create database hive DEFAULT CHARSET latin1;

--集群监控数据库
create database amon DEFAULT CHARSET utf8;

--hue数据库
create database hue DEFAULT CHARSET utf8;

--oozie数据库
create database oozie default charset utf8;

--授权
grant all privileges on hive.* to 'scm'@'%' identified by 'cdh' with grant option;

grant all privileges on amon.* to 'scm'@'%' identified by 'cdh' with grant option;

grant all privileges on hue.* to 'scm'@'%' identified by 'cdh' with grant option;

grant all privileges on oozie.* to 'scm'@'%' identified by 'cdh' with grant option;

--远程访问->

grant all privileges on *.* to 'root'@'%' identified by '123456' with grant option;

flush privileges; 

4.主节点解压安装cloudera-manager-el6-cm5.13.3_x86_64.tar.gz 
cloudera manager的目录默认位置在/opt下,解压:

tar -zxvf cloudera-manager-*.tar.gz;

将解压后的目录放到/opt目录下。

5.为Cloudera Manager 5建立数据库

去MySql的官网下载JDBC驱动,找到mysql-connector-java-5.1.43-bin.jar,放到/opt/cm-5.13.3/share/cmf/lib/中

6.初始化cloudera manager数据库配置 
/opt/cm-5.13.3/share/cmf/schema/scm_prepare_database.sh -h <mysql所在主机> mysql cm scm cdh 

注意:上行命令更正为:

/opt/cm-5.13.3/share/cmf/schema/scm_prepare_database.sh -h <mysql所在主机> mysql -uroot -p123456 cm scm cdh

三、其他配置

1.在所有节点创建cloudera-scm用户

useradd --system --home=/opt/cm-5.13.3/run/cloudera-scm-server/ --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

2.修改/opt/cm-5.13.3/etc/cloudera-scm-agent/config.ini中的server_host为主节点的主机名。

vim /opt/cm-5.13.3/etc/cloudera-scm-agent/config.ini

3.同步Agent到其他节点

scp -r /opt/cm-5.13.3 bigdata1:/opt/
scp -r /opt/cm-5.13.3 bigdata2:/opt/
scp -r /opt/cm-5.13.3 bigdata3:/opt/
scp -r /opt/cm-5.13.3 bigdata4:/opt/

4.准备Parcels,用以安装CDH5

将CHD5相关的Parcel包放到主节点的/opt/cloudera/parcel-repo/目录中 
相关的文件如下: 
CDH-5.13.3-1.cdh5.13.3.p0.2-el6.parcel 
CDH-5.13.3-1.cdh5.13.3.p0.2-el6.parcel.sha1 
manifest.json

最后将CDH-5.13.3-1.cdh5.13.3.p0.2-el6.parcel.sha1,重命名为CDH-5.13.3-1.cdh5.13.3.p0.2-el6.parcel.sha,这点必须注意,否则,系统会重新下载文件。

5.相关启动脚本

/opt/cm-5.13.3/etc/init.d/cloudera-scm-server start 
/opt/cm-5.13.3/etc/init.d/cloudera-scm-agent start 
<主节点都启动、其他节点只启动agent>

5.哦了,等一会打开浏览器 
http://bigdata1:7180 
按照步骤开始配置就行了

参考:https://blog.csdn.net/ltliyue/article/details/82734814
 

猜你喜欢

转载自blog.csdn.net/sinat_36710456/article/details/84025089