Centos7在线安装CDH6

以下步骤亲测有效无误！！！！！

一、 基础环境准备

1）软件清单
Centos7(64位)服务器3台，地址如下：
jdk1.8
Mysql5.7 （必须要5.7的版本！！）
mysql-connector-java-5.1.42-bin.jar
cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm（https://archive.cloudera.com/cm6/6.3.1/redhat7/yum/RPMS/x86_64/）
cloudera-manager-daemons-6.2.0-968826.el7.x86_64.rpm（https://archive.cloudera.com/cm6/6.3.1/redhat7/yum/RPMS/x86_64/）
cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm（https://archive.cloudera.com/cm6/6.3.1/redhat7/yum/RPMS/x86_64/）
oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm（https://archive.cloudera.com/cm6/6.3.1/redhat7/yum/RPMS/x86_64/）
CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel（https://archive.cloudera.com/cdh6/6.3.2/parcels/）
CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1（https://archive.cloudera.com/cdh6/6.3.2/parcels/）
manifest.json（https://archive.cloudera.com/cdh6/6.3.2/parcels/）

2）服务器地址
192.168.56.1 （此地址映射你自己的服务器地址）
192.168.56.2 （此地址映射你自己的服务器地址）
192.168.56.3 （此地址映射你自己的服务器地址）

3）修改hostname

vim /etc/hostname

修改主机名为cdh*后进行以下操作

service network restart

4）配置hosts

vim /etc/hosts

192.168.56.1 cdh1
192.168.56.2 cdh2
192.168.56.3 cdh3

5）免密访问

ssh-keygen -t rsa
//三台服务器全都执行以下命令
ssh-copy-id  root@cdh1
ssh-copy-id  root@cdh2
ssh-copy-id  root@cdh1

6）关闭防火墙

systemctl stop firewalld
systemctl disable firewalld

7）关闭SELINUX

setenforce 0
vim /etc/sysconfig/selinux
SELINUX=disabled

8）配置NTP

yum -y install ntp
systemctl start ntpd 
systemctl enable ntpd 
在cdh1节点
vim /etc/ntp.conf
server 0.cn.pool.ntp.org
server 1.cn.pool.ntp.org
server 2.cn.pool.ntp.org
server 3.cn.pool.ntp.org
restrict 0.cn.pool.ntp.org nomodify notrap noquery
restrict 1.cn.pool.ntp.org nomodify notrap noquery
restrict 2.cn.pool.ntp.org nomodify notrap noquery
restrict 3.cn.pool.ntp.org nomodify notrap noquery
server 127.127.1.0
fudge 127.127.1.0 stratum 10

systemctl start ntpd.service
systemctl enable ntpd.service
ntpstat
在cdh[2-3]节点
vim /etc/ntp.conf
server cdh1 prefer

systemctl start ntpd.service
systemctl enable ntpd.service
ntpdate -u cdh1
ntpstat

9）安装JDK

mkdir /usr/local/java
cp /home/ jdk-8u131-linux-x64.gz /usr/local/java/
cd /usr/local/java/
tar xvf jdk-8u131-linux-x64.gz
rm jdk-8u131-linux-x64.gz
scp -r /usr/local/java/ root@cdh2:/usr/local/
scp -r /usr/local/java/ root@cdh3:/usr/local/

设置环境变量

vi /etc/profile
export JAVA_HOME=/usr/local/jdk
export JRE_HOME=$JAVA_HOME/jre
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$PATH

java -version

10）安装MySQL

删除自带数据库

rpm -qa | grep mariadb
rpm -e –nodeps 包名
rm -rf /etc/my.cnf

下载依赖并创建用户和组

yum -y install perl perl-devel autoconf
groupadd mysql
useradd -g mysql mysql
id mysql

安装请参考以下链接：

https://blog.csdn.net/baidu_32872293/article/details/80557668
https://www.cnblogs.com/downmoon/p/11896158.html

设置数据库允许远程登陆

mysql -u root -p
use mysql;
update user set host=’%’ where user=’root’;

若报错可能是已经设置好了，用下列语句查看

select host from user where user = ‘root’;
flush privileges;

更改数据库字符驱动

set character_set_client=utf8;
set character_set_connection=utf8;
set character_set_results=utf8;
set character_set_server=utf8;
set character_set_system=utf8;

查看修改结果

show variables like ‘character_set_%’;

关闭数据库安全模式，使数据库可以增删改

SET SQL_SAFE_UPDATES=0;

创建数据库并使本地root用户有访问权限

mysql -u root -p
create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database am DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database oozie DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
grant all privileges on *.* to ‘root’@’cdh1’ identified by ‘000000’ with grant option;
flush privileges;

安装CDH Server（master节点）

提前准备：

将mysql-connector-java-5.1.42-bin.jar改名为mysql-connector-java.jar放入/usr/share/java目录下

数据库修改/etc/my.cnf

[mysqld]
validate_password=off
wait_timeout=31536000
interactive_timeout=31536000

安装cloudera的rpm

yum install oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm

yum install cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm

parcel初始化
新建parcel目录

mkdir -p /opt/cloudera/parcel-repo

将安装包文件复制到默认安装路径

cp /home/CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel /opt/cloudera/parcel-repo/
cp /home/CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1 /opt/cloudera/parcel-repo/CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha
cp /home/manifest.json /opt/cloudera/parcel-repo/

执行创建数据库命令

/opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm -hlocalhost  -uroot -p'000000' --scm-host localhost scm scm scm

重启cloudera-scm-server

service cloudera-scm-server restart

地址：http://localhost:7180
用户：admin
密码：admin

安装CDH Agent(每个节点)

提前准备：

将mysql-connector-java-5.1.42-bin.jar改名为mysql-connector-java.jar放入/usr/share/java目录下

安装cloudera的rpm

yum install oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm

yum install  cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm

修改agent配置

vim /etc/cloudera-scm-agent/config.ini
server_host=cdh1

启动agent

systemctl start cloudera-scm-agent

常见问题汇总

1、检查主机正确性警告解决方式

在三台服务器中执行以下命令

sysctl -w vm.swappiness=10
vim /etc/sysctl.conf
vm.swappiness=10

echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
vim /etc/rc.local
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled

2、数据库设置hue数据库无法连接

在cdh2和cdh3节点执行一下命令

scp -r /usr/lib64/mysql/libmysqlclient.so.18 root@cdh1:/usr/lib64

或者参考
https://blog.csdn.net/ckg8933/article/details/83379279

3、hive缺少mysql驱动

cdh1执行

cp /home/mysql-connector-java-5.1.42-bin.jar /opt/cloudera/parcels/CDH-5.8.5-1.cdh5.8.5.p0.5/lib/hive/lib/

4、oozie缺少mysql驱动

cdh1执行

cp /home/mysql-connector-java-5.1.42-bin.jar /opt/cloudera/parcels/CDH-5.8.5-1.cdh5.8.5.p0.5/lib/oozie/lib/
cp /home/mysql-connector-java-5.1.42-bin.jar /var/lib/oozie/mysql-connector-jave.jar
chmod 777 /var/lib/oozie/mysql-connector-jave.jar

5、hue数据库测试连接失败

https://blog.csdn.net/ckg8933/article/details/83379279

6、数据库测试连接失败

https://blog.csdn.net/levy_cui/article/details/51142843

7、找不到本地parcle库

chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo/*
chmod 777 /opt/cloudera/parcel-repo/*

重启服务

8、hdfs文件夹权限问题

chmod 777 -R nn
chmod 777 -R snn
chmod o-w -R dn
chmod o-w -R var

9、缺少libmysqlclient.so.18()

下载安装此组件
https://download.csdn.net/download/lufengbo123/12195518

10、hbase使用中出现各种异常

删除zookeeper下hbase目录，然后重启hbase即可

如果还是出现Master is initializing报错，则在zookeeper中删除hbase目录后，停止hbase集群，使用 hdfs dfs -rmr /hbase即可
删除hdfs文件后，在hbase前端新建hbase根目录然后重启即可。

11、启动前

看看各个组件内部是否需要创建文件夹和数据库

12、hue中使用hbase报错

Hbase中取消以下两个选项

hue中添加以下选项
[hbase]
thrift_transport=buffered