hadoop生态圈框架工具甚多，自己整合起来很是繁琐，特别是oozie以及hue结合来实现hive、mr、spark等定时依赖任务更是一步一个坑，为了减少踩坑，只好从apache hadoop，转向CDH。

两个各有利弊，CDH贵在整合方便。本次教程大部分来自http://www.cnblogs.com/fujiangong/p/5620050.html，稍作修改，记录踩下的坑。

JDK环境

JDK版本：jdk-8u121-linux-x64.rpm

CM包

CM版本：5.10.0 cloudera-manager-centos7-cm5.10.0_x86_64.tar.gz

下载地址：http://archive.cloudera.com/cm5/cm/5/cloudera-manager-centos7-cm5.10.0_x86_64.tar.gz

CDH包

CDH版本：5.10.0

CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel.sha1 manifest.json

下载地址

http://archive.cloudera.com/cdh5/parcels/5.10.0/CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel

http://archive.cloudera.com/cdh5/parcels/5.10.0/CDH-5.10.0-1.cdh5.10.0.p0.41-el7.parcel.sha1

http://archive.cloudera.com/cdh5/parcels/5.10.0/manifest.json

java-mysql连接jar包

下载地址：http://central.maven.org/maven2/mysql/mysql-connector-java/5.1.22/mysql-connector-java-5.1.22.jar

备注：mysql的jar包不要使用6.x版本，会有兼容问题，这里使用的是5.1.22

数据库

类型：MariaDB

节点

10.0.0.1 slave1
10.0.0.2 slave2
10.0.0.3 slave3
10.0.0.4 slave4
10.0.0.5 master
10.0.0.6 standby

二、底层环境准备

注：准备安装前需要把相关的包文件下载下来，并上传到服务器上

1、安装JDK

所有节点

1）安装系统原有java

yum -y install java

2）卸载OpenJDK

cdh指定使用oracle jdk，所以卸载掉

openjdk-headless

openjdk

保留其它工具包

3）安装下载的java

rpm -ivh jdk-8u121-linux-x64.rpm

2、更改主机名和hosts

所有节点

注意每个主机名对应的IP，按实际情况更改。

/etc/hostname

/etc/hosts

3、关闭防火墙

systemctl stop firewalld

systemctl disable firewalld

sed -i "s/SELINUX=enforcing/SELINUX=disabled/" /etc/selinux/config

reboot

4、安装NTP服务

所有节点

yum -y install ntp

配置略过

5、SSH无密码登陆

配置Cloudera Management Service所在节点无密码登陆到其他节点

本次在节点standby节点

ssh-keygen -t rsa

Generating public/private rsa key pair.

Enter file in which to save the key (/root/.ssh/id_rsa): 回车

Enter passphrase (empty for no passphrase): 回车

Enter same passphrase again: 回车

Your identification has been saved in /root/.ssh/id_rsa.

Your public key has been saved in /root/.ssh/id_rsa.pub.

The key fingerprint is:

bc:14:e2:59:42:1f:21:80:eb:49:d8:3b:7c:8d:b7:f9 [email protected]

The key's randomart image is:

+--[ RSA 2048]----+

| ...o o. |

| . . o . |

| o . o + |

|. + . * . |

| + o oo S |

| * o o. . |

| o . o. |

| o |

| .E |

+-----------------+

ssh-copy-id master

ssh-copy-id slave1

ssh-copy-id slave2

ssh-copy-id slave3

ssh-copy-id slave4

6、安装数据库

service节点

yum -y install mariadb-servervi /etc/my.cnf

[mysqld]

datadir=/var/lib/mysql

socket=/var/lib/mysql/mysql.sock

symbolic-links=0

character-set-server=utf8 \\添加的行

[mysqld_safe]

log-error=/var/log/mariadb/mariadb.log

pid-file=/var/run/mariadb/mariadb.pidsystemctl start mariadb

systemctl enable mariadb

初始化数据库

mysql_secure_installation/usr/bin/mysql_secure_installation: line 379: find_mysql_client: command not found

NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB

SERVERS IN PRODUCTION USE! PLEASE READ EACH STEP CAREFULLY!

In order to log into MariaDB to secure it, we'll need the current

password for the root user. If you've just installed MariaDB, and

you haven't set the root password yet, the password will be blank,

so you should just press enter here.

Enter current password for root (enter for none):

OK, successfully used password, moving on...

Setting the root password ensures that nobody can log into the MariaDB

root user without the proper authorisation.

Set root password? [Y/n] y \\设置数据库root密码

New password: \\设置密码

Re-enter new password: \\再次输入密码

Password updated successfully!

Reloading privilege tables..

... Success!

By default, a MariaDB installation has an anonymous user, allowing anyone

to log into MariaDB without having to have a user account created for

them. This is intended only for testing, and to make the installation

go a bit smoother. You should remove them before moving into a

production environment.

Remove anonymous users? [Y/n] y \\移除匿名用户

... Success!

Normally, root should only be allowed to connect from 'localhost'. This

ensures that someone cannot guess at the root password from the network.

Disallow root login remotely? [Y/n] y \\禁止root用户远程登录

... Success!

By default, MariaDB comes with a database named 'test' that anyone can

access. This is also intended only for testing, and should be removed

before moving into a production environment.

Remove test database and access to it? [Y/n] y \\删除测试数据库

- Dropping test database...

... Success!

- Removing privileges on test database...

... Success!

Reloading the privilege tables will ensure that all changes made so far

will take effect immediately.

Reload privilege tables now? [Y/n] //

... Success!

Cleaning up...

All done! If you've completed all of the above steps, your MariaDB

installation should now be secure.

Thanks for using MariaDB!

7、其他安装与配置

所有节点

# yum -y install psmisc MySQL-python at bc bind-libs bind-utils cups-client cups-libs cyrus-sasl-gssapi cyrus-sasl-plain ed fuse fuse-libs httpd httpd-tools keyutils-libs-devel krb5-devel libcom_err-devel libselinux-devel libsepol-devel libverto-devel mailcap noarch mailx mod_ssl openssl-devel pcre-devel postgresql-libs python-psycopg2 redhat-lsb-core redhat-lsb-submod-security x86_64 spax time zlib-devel

# chmod +x /etc/rc.d/rc.local

# echo "echo 0 > /proc/sys/vm/swappiness" >>/etc/rc.d/rc.local

# echo "echo never > /sys/kernel/mm/transparent_hugepage/defrag" >>/etc/rc.d/rc.local

# echo 0 > /proc/sys/vm/swappiness

# echo never > /sys/kernel/mm/transparent_hugepage/defrag

# yum -y install rpcbind

# systemctl start rpcbind

# echo "systemctl start rpcbind" >> /etc/rc.d/rc.local

三、开始安装CDH

1、传宝、解包

把下载好的一下相应包文件上传到service节点内

mysql-java连接包在所有节点上都要上传

tar zxvf cloudera-manager-centos7-cm5.10.0_x86_64.tar.gz -C /opt/

2、创建数据库

# mysql -uroot -p

Enter password: \\输入数据库密码

Welcome to the MariaDB monitor. Commands end with ; or \g.

Your MariaDB connection id is 10

Server version: 5.5.47-MariaDB MariaDB Server

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;