基于centos7.2的CDH5.7.2的部署

1 本次是基于centos7.2的CDH5.7.2的部署，希望能给大家做个参考

2 前置说明

2.1 服务器的配置说明
       系统：建议使用centos7.x

2.2 CDH架构平台

2.3 本次搭建软件和角色说明

使用了3台服务器

系统使用的是：centos7.2
JDK使用的：jdk-7u79-linux-x64.tar.gz
CDH：  cloudera-manager-centos7-cm5.7.2_x86_64.tar.gz
       CDH-5.7.2-1.cdh5.7.2.p0.18-el7.parcel
       CDH-5.7.2-1.cdh5.7.2.p0.18-el7.parcel.sha
       manifest.json
依赖包：mysql-connector-java-5.1.31-bin.jar

所扮演的角色如下：

                   Server              Agent
hadoop01             1                   1
hadoop02                                 1
hadoop03                                 1

3 CDH的部署

3.1 基础环境准备

3.1.1 主机名映射(全部节点)

vi /etc/hosts 如下：
192.168.5.221 hadoop01
192.168.5.222 hadoop02
192.168.5.230 hadoop03

验证：

3.1.2 关闭防火墙(全部节点)

iptables -X
iptables -F
iptables -Z

systemctl stop firewalld.service
systemctl disable firewalld.service
firewall-cmd --state

vi /etc/selinux/config
改：SELINUX=permissive 如下

reboot

3.1.3 yum的安装

注：每个基于系统的部署不同yum的配置也有所不同。下面的一种只能作为参考用：
首先上传系统的镜像CentOS-7-x86_64-DVD-1511.iso 放在/home/tools

hadoop01节点的配置:

cd /etc/yum.repos.d/
rm -rf *
vi local.repo
[centos]
name=centos
baseurl=file:///opt/centos
gpgcheck=0
enabled=1

mkdir /opt/centos
mount -o loop /home/tools/CentOS-7-x86_64-DVD-1511.iso /opt/centos/
yum clean all
yum list

安装ftp服务器
yum install vsftpd –y
vi /etc/vsftpd/vsftpd.conf
后面加多一行anon_root=/opt---引导ftp到/opt目录下

systemctl restart vsftpd
chkconfig vsftpd on

其它节点的配置如下：
cd /etc/yum.repos.d/
rm -rf *

vi local.repo
[centos]
name=centos
baseurl=ftp://hadoop01/centos
gpgcheck=0
enabled=1

yum clean all
yum list

3.1.4 安装openssh-clients (全部节点)

yum install openssh-clients –y

3.1.5 ntp时间同步

所有节点：yum install ntp -y

hadoop01节点执行
vi /etc/ntp.conf
添加如下：
server 127.127.1.0
fudge 127.127.1.0 stratum 10

systemctl restart ntpd
chkconfig ntpd on

其他节点执行 
ntpdate hadoop01

chkconfig ntpd on

3.2 CDH部署基于环境准备

3.2.1 安装免秘钥(全部节点都要实现)

所有的节点>>对所有的节点免秘钥
ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
cd /root/.ssh/
scp id_dsa.pub hadoop02:/tmp/
cat /tmp/id_dsa.pub >> /root/.ssh/authorized_keys
如此类推
ssh hadoop02
验证：
 

3.2.2 JDK的部署(全部节点)

mkdir /home/tools
scp jdk-7u79-linux-x64.tar.gz @hadoop02:/home/tools/.
mkdir /usr/java
tar zxvf jdk-7u79-linux-x64.tar.gz -C /usr/java/.

vi /etc/profile
export JAVA_HOME=/usr/java/jdk1.7.0_79
export PATH=$PATH:$JAVA_HOME/bin

source /etc/profile
java –version
验证如下表明安装成功：

3.2.3 安装MySql(Server节点)

rpm -ivh mysql-community-common-5.7.22-1.el7.x86_64.rpm
rpm -ivh mysql-community-libs-5.7.22-1.el7.x86_64.rpm
rpm -ivh mysql-community-devel-5.7.22-1.el7.x86_64.rpm
rpm -ivh mysql-community-libs-compat-5.7.22-1.el7.x86_64.rpm
rpm -ivh mysql-community-client-5.7.22-1.el7.x86_64.rpm
rpm -ivh mysql-community-server-5.7.22-1.el7.x86_64.rpm --force -nodeps
或者
rpm -ivh mysql-community-* --force --nodeps

service mysqld stop
vi /etc/my.cnf
skip-grant-tables
sql_mode=STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION

service mysqld start
chkconfig mysqld on

mysql -u root –p #不用密码直接回车
set password for root@localhost = password('000000');
flush privileges;
set password for root@localhost = password(''000000');
flush privileges;
grant all privileges on *.* to root@"%" identified by "000000" with grant option;
flush privileges;
quit;

验证如下：
mysql -u root –p

service mysqld stop
vi /etc/my.cnf
#skip-grant-tables

service mysqld start
systemctl enable mysqld

3.2.4 安装第三方依赖包(全部节点)

yum install chkconfig -y
yum install python -y
yum install bind-utils -y
yum install psmisc -y
yum install libxslt -y
yum install zlib -y
yum install sqlite -y
yum install cyrus-sasl-plain -y
yum install cyrus-sasl-gssapi -y
yum install fuse -y
yum install fuse-libs -y
yum install redhat-lsb –y

或者：
yum install chkconfig python bind-utils psmisc libxslt zlib sqlite cyrus-sasl-plain cyrus-sasl-gssapi fuse fuse-libs redhat-lsb –y

3.3 CDH的正式部署

3.3.1 解压安装Cloudera Manager Server、Agent(全部节点)

mkdir /opt /cloudera-manager
cd /home/tools
scp cloudera-manager-centos7-cm5.7.2_x86_64.tar.gz  hadoop02:/home/tools/.
tar zxvf cloudera-manager-centos7-cm5.7.2_x86_64.tar.gz  -C  /opt/cloudera-manager

3.3.2 创建用户cloudera-scmt(全部节点)

useradd --system --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

3.3.3 配置CM Agent(全部节点)

修改文件config.ini中 server_host
cd /opt /cloudera-manager/cm-5.7.2/etc/cloudera-scm-agent/
vi config.ini
server_host=hadoop01

3.3.4 配置CM Server数据库(Server节点)

拷贝mysql jar文件到目录 /usr/share/java/
注意jar包名称要修改为mysql-connector-java.jar

mkdir /usr/share/java
cd /usr/share/java
cp /home/tools/mysql-connector-java-5.1.31-bin.jar  .
mv mysql-connector-java-5.1.31-bin.jar  mysql-connector-java.jar

进入mysql:
mysql -u root –p

GRANT ALL PRIVILEGES ON *.* TO 'temp'@'%' IDENTIFIED BY 'temp' WITH GRANT OPTION;

cd /opt/cloudera-manager/cm-5.7.2/share/cmf/schema/
./scm_prepare_database.sh mysql temp -h hadoop01 -utemp -ptemp --scm-host hadoop01 scm scm scm

注：如果报错则：
vi /etc/my.cnf
validate-password=OFF

service mysqld restart

3.3.5 创建Parcel目录

Server节点
mkdir -p /opt /cloudera/parcel-repo
chown cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo

Agent节点
mkdir -p /opt/cloudera/parcels
chown cloudera-scm:cloudera-scm /opt /cloudera/parcels

3.3.6 制作CDH本地源

下载好文件CDH-5.7.2-1.cdh5.7.2.p0.18-el7.parcel, CDH-5.7.2-1.cdh5.7.2.p0.18-el7.parcel.sha以及manifest.json，将这两个文件放到server节点的/opt /cloudera /parcel-repo下。
打开manifest.json文件，里面是json格式的配置，找到与下载版本相对应的hash码，新建文件，文件名与你的parel包名一致，并加上.sha后缀，将hash码复制到文件中保存

cd /opt /cloudera/parcel-repo/
cp /home/tools/cdh7/CDH-5.7.2-1.cdh5.7.2.p0.18-el7.parcel.sha .
cp /home/tools/cdh7/CDH-5.7.2-1.cdh5.7.2.p0.18-el7.parcel .
cp /home/tools/cdh7/manifest.json .

3.3.7 启动CM Server、Agent

Server节点
cd /opt/cloudera-manager/cm-5.7.2/etc/init.d/
./cloudera-scm-server start

Agent所有节点
cd /opt/cloudera-manager/cm-5.7.2/etc/init.d/
./cloudera-scm-agent start

在Server节点查看运行状态
cd ../..
cd log/
tail -f cloudera-scm-server.log

3.3.8 访问和检查CDH

访问：http:/hadoop01:7180
用户名、密码：admin
若可以访问，则CM安装成功。

3.4 CDH的组件部署

3.4.1 基础界面的部署

这个需要时间等待……

全部节点都要
echo 0 > /proc/sys/vm/swappiness
cat /proc/sys/vm/swappiness

然后刷新界面

全部节点都要
vi /etc/rc.local
echo never >/sys/kernel/mm/transparent_hugepage/defrag

3.4.2 HDFS，YARN，ZooKeeper的部署

3.4.3 重启检查启动(如果集群启动正常可不做)

3.4.4 Hive的部署

进入MySql库：
create database hive  DEFAULT CHARACTER SET utf8;
grant all on hive.* TO 'hive'@'%' IDENTIFIED BY 'hive';

3.4.5 Sqoop的部署

3.4.6 Impala的部署

3.4.7 Spark的部署

3.4.8 Oozie，Hue的部署

Oozie的部署：

hadoop01中创建数据库：
create database oozie  DEFAULT CHARACTER SET utf8;
grant all on oozie.* TO 'oozie'@'%' IDENTIFIED BY 'oozie';

hadoop02的服务器上上传mysql-connector-java-5.1.31-bin.jar：

且执行如下命令

mkdir /usr/share/java

cd /usr/share/java

cp /home/tools/mysql-connector-java-5.1.31-bin.jar  .

mv mysql-connector-java-5.1.31-bin.jar  mysql-connector-java.jar

Hue的部署：