Hadoop-CDH5.7.0 for CentOS7

一、需求

系统

CentOS 7 最小化安装

yum -y install psmisc MySQL-python at bc bind-libs bind-utils cups-client cups-libs cyrus-sasl-gssapi cyrus-sasl-plain ed fuse fuse-libs httpd httpd-tools keyutils-libs-devel krb5-devel libcom_err-devel libselinux-devel libsepol-devel libverto-devel mailcap noarch mailx mod_ssl openssl-devel pcre-devel postgresql-libs python-psycopg2 redhat-lsb-core redhat-lsb-submod-security x86_64 spax time zlib-devel

yum install krb5-devel cyrus-sasl-gssapi cyrus-sasl-deve libxml2-devel libxslt-devel mysql mysql-devel openldap-devel python-devel python-simplejson sqlite-devel

yum -y install perl

yum install报错[Errno -1] Package does not match intended download

的解决办法:

yum clean all

yum makecache

之后再试。

JDK环境

JDK版本：jdk-8u191-linux-x64.tar.gz

CM包

CM版本：5.7.0 cloudera-manager-centos7-cm5.7.0_x86_64.tar.gz

下载地址：http://archive.cloudera.com/cm5/cm/5/cloudera-manager-centos7-cm5.7.0_x86_64.tar.gz

CDH包

CDH版本：5.7.0

CDH-5.7.0-1.cdh5.7.0.p0.45-el7.parcel CDH-5.7.0-1.cdh5.7.0.p0.45-el7.parcel.sha1 manifest.json

下载地址

http://archive.cloudera.com/cdh5/parcels/5.7/CDH-5.7.1-1.cdh5.7.1.p0.11-el7.parcel

http://archive.cloudera.com/cdh5/parcels/5.7/CDH-5.7.1-1.cdh5.7.1.p0.11-el7.parcel.sha1

http://archive.cloudera.com/cdh5/parcels/5.7/manifest.json

java-mysql连接jar包

下载地址：http://central.maven.org/maven2/mysql/mysql-connector-java/8.0.13/mysql-connector-java-8.0.13.jar

数据库

类型：MariaDB

版本：5.5.47-MariaDB MariaDB Server

节点

主机名	IP地址	说明
bigdata01	172.17.28.1	master datanode namenode
bigdata02	172.17.28.2	datanode
bigdata03	172.17.28.3	datanode
bigdata04	172.17.28.4	datanode

节点的数量可以根据自己的需求变更，也可以在CDH安装完成后，在管理界面自己拓展添加

二、底层环境准备

注：准备安装前需要把相关的包文件下载下来，并上传到服务器上

1、安装JDK

所有节点

1）安装系统原有java

# yum -y install java

# java -version

openjdk version "1.8.0_91"

OpenJDK Runtime Environment (build 1.8.0_91-b14)

OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode)

# rpm -qa|grep java

javapackages-tools-3.4.1-11.el7.noarch

python-javapackages-3.4.1-11.el7.noarch

java-1.8.0-openjdk-headless-1.8.0.91-0.b14.el7_2.x86_64

java-1.8.0-openjdk-1.8.0.91-0.b14.el7_2.x86_64

tzdata-java-2016d-1.el7.noarch

如果操作系统已安装openJDK则卸载

2）卸载OpenJDK

# rpm -e --nodeps java-1.8.0-openjdk-headless-1.8.0.91-0.b14.el7_2.x86_64

# rpm -e --nodeps java-1.8.0-openjdk-1.8.0.91-0.b14.el7_2.x86_64

3）安装下载的java

解压缩到/usr/java/jdk1.8.0_191

配置环境变量:

在 /etc/profile文件最后增加

JAVA_HOME=/usr/java/jdk1.8.0_191

PATH=$JAVA_HOME/bin:$PATH

CLASSPATH=$JAVA_HOME/jre/lib/ext:$JAVA_HOME/lib/tools.jar

export PATH JAVA_HOME CLASSPATH

保存后执行：source /etc/profile，使环境变量立即生效。

# java -version

java version "1.8.0_91"

Java(TM) SE Runtime Environment (build 1.8.0_91-b14)

Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)

# javac -version

javac 1.8.0_91

2、更改主机名和hosts

所有节点

注意每个主机名对应的IP，按实际情况更改。

# echo "bigdata0*" > /etc/hostname把各个自己名写入

# echo "172.17.28.1 bigdata01" >> /etc/hosts

# echo "172.17.28.2 bigdata02" >> /etc/hosts

# echo "172.17.28.3 bigdata03" >> /etc/hosts

# echo "172.17.28.4 bigdata04" >> /etc/hosts

3、关闭防火墙

所有节点

# systemctl stop firewalld

# systemctl disable firewalld

Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.

Removed symlink /etc/systemd/system/basic.target.wants/firewalld.service.

# systemctl status firewalld

● firewalld.service - firewalld - dynamic firewall daemon

Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)

Active: inactive (dead)

Jun 20 16:05:43 bigdata05 systemd[1]: Starting firewalld - dynamic firewall daemon...

Jun 20 16:05:45 bigdata05 systemd[1]: Started firewalld - dynamic firewall daemon.

Jun 20 16:17:51 bigdata05 systemd[1]: Stopping firewalld - dynamic firewall daemon...

Jun 20 16:17:52 bigdata05 systemd[1]: Stopped firewalld - dynamic firewall daemon.

# sed -i "s/SELINUX=enforcing/SELINUX=disabled/" /etc/selinux/config

# reboot

重启服务器使hostname和selinux的配置生效

# getenforce

Disabled

# sestatus -v

SELinux status: disabled

4、安装NTP服务

所有节点

# yum -y install ntp

更改配置文件/etc/ntp.conf

在master节点namenode节点

注释掉原有的server指向把时间服务器的域名地址指向一个靠谱可连的时间服务器，我选择的是ntp.api.bz.

# vi /etc/ntp.conf

……

#server 0.centos.pool.ntp.org iburst

#server 1.centos.pool.ntp.org iburst

#server 2.centos.pool.ntp.org iburst

#server 3.centos.pool.ntp.org iburst

server ntp.api.gz iburst

......

在其余的节点上，把时间服务器的server指向master namenode节点

# vi /etc/ntp.conf

……

#server 0.centos.pool.ntp.org iburst

#server 1.centos.pool.ntp.org iburst

#server 2.centos.pool.ntp.org iburst

#server 3.centos.pool.ntp.org iburst

server 172.17.28.1 iburst

......

所有节点

# systemctl start ntpd

# systemctl enable ntpd

Created symlink from /etc/systemd/system/multi-user.target.wants/ntpd.service to /usr/lib/systemd/system/ntpd.service.

立即同步时间

ntpdate -u 202.112.10.36

把系统时间写入硬件

hwclock -w

5、SSH无密码登陆

配置master namenode节点无密码登陆到其他节点

在master namenode节点

# ssh-keygen -t rsa

Generating public/private rsa key pair.

Enter file in which to save the key (/root/.ssh/id_rsa): 回车

Enter passphrase (empty for no passphrase): 回车

Enter same passphrase again: 回车

Your identification has been saved in /root/.ssh/id_rsa.

Your public key has been saved in /root/.ssh/id_rsa.pub.

The key fingerprint is:

bc:14:e2:59:42:1f:21:80:eb:49:d8:3b:7c:8d:b7:f9 root@bigdata05

The key's randomart image is:

+--[ RSA 2048]----+

| ...o o. |

| . . o . |

| o . o + |

|. + . * . |

| + o oo S |

| * o o. . |

| o . o. |

| o |

| .E |

+-----------------+

# ssh-copy-id 172.17.28.2

# ssh-copy-id 172.17.28.3

# ssh-copy-id 172.17.28.4

6、安装数据库

master namenode节点

# yum -y install mariadb-server

# vi /etc/my.cnf

[mysqld]

datadir=/var/lib/mysql

socket=/var/lib/mysql/mysql.sock

symbolic-links=0

character-set-server=utf8 \\添加的行

[mysqld_safe]

log-error=/var/log/mariadb/mariadb.log

pid-file=/var/run/mariadb/mariadb.pid !includedir /etc/my.cnf.d

# systemctl start mariadb

# systemctl enable mariadb ln -s '/usr/lib/systemd/system/mariadb.service' '/etc/systemd/system/multi-user.target.wants/mariadb.service'

# mysql_secure_installation /usr/bin/mysql_secure_installation: line 379: find_mysql_client: command not found

NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB

SERVERS IN PRODUCTION USE! PLEASE READ EACH STEP CAREFULLY!

In order to log into MariaDB to secure it, we'll need the current

password for the root user. If you've just installed MariaDB, and

you haven't set the root password yet, the password will be blank,so you should just press enter here.

Enter current password for root (enter for none):

OK, successfully used password, moving on...

Setting the root password ensures that nobody can log into the MariaDB

root user without the proper authorisation.

Set root password? [Y/n] y \\设置数据库root密码

New password: \\设置密码

Re-enter new password: \\再次输入密码

Password updated successfully!

Reloading privilege tables..

... Success!

By default, a MariaDB installation has an anonymous user, allowing anyone

to log into MariaDB without having to have a user account created for

them. This is intended only for testing, and to make the installation

go a bit smoother. You should remove them before moving into a

production environment.

Remove anonymous users? [Y/n] y \\移除匿名用户

... Success!

Normally, root should only be allowed to connect from 'localhost'. This

ensures that someone cannot guess at the root password from the network.

Disallow root login remotely? [Y/n] y \\禁止root用户远程登录

... Success!

By default, MariaDB comes with a database named 'test' that anyone can

access. This is also intended only for testing, and should be removed

before moving into a production environment.

Remove test database and access to it? [Y/n] y \\删除测试数据库

- Dropping test database...

... Success!

- Removing privileges on test database...

... Success!

Reloading the privilege tables will ensure that all changes made so far

will take effect immediately.

Reload privilege tables now? [Y/n] // ... Success!

Cleaning up...

All done! If you've completed all of the above steps, your MariaDBinstallation should now be secure.

Thanks for using MariaDB!

数据库只是安装了一下，并没有做一些详细的配置。如果有需求可以另行配置。

7、其他安装与配置

所有节点

# yum -y install psmisc MySQL-python at bc bind-libs bind-utils cups-client cups-libs cyrus-sasl-gssapi cyrus-sasl-plain ed fuse fuse-libs httpd httpd-tools keyutils-libs-devel krb5-devel libcom_err-devel libselinux-devel libsepol-devel libverto-devel mailcap noarch mailx mod_ssl openssl-devel pcre-devel postgresql-libs python-psycopg2 redhat-lsb-core redhat-lsb-submod-security x86_64 spax time zlib-devel

# chmod +x /etc/rc.d/rc.local

# echo "echo 0 > /proc/sys/vm/swappiness" >>/etc/rc.d/rc.local

# echo "echo never > /sys/kernel/mm/transparent_hugepage/defrag" >>/etc/rc.d/rc.local

# echo 0 > /proc/sys/vm/swappiness

# echo never > /sys/kernel/mm/transparent_hugepage/defrag

# yum -y install rpcbind

# systemctl start rpcbind

# echo "systemctl start rpcbind" >> /etc/rc.d/rc.local

三、开始安装CDH

1、传宝、解包

把下载好的一下相应包文件上传到master namenode节点内

mysql-java连接包在所有节点上都要上传

# ls

CDH-5.7.0-1.cdh5.7.0.p0.45-el7.parcel cloudera-manager-centos7-cm5.7.0_x86_64.tar.gz manifest.json

CDH-5.7.0-1.cdh5.7.0.p0.45-el7.parcel.sha1 jdk-8u91-linux-x64.rpm mysql-connector-java-8.0.13.jar

# tar zxvf cloudera-manager-centos7-cm5.7.0_x86_64.tar.gz -C /opt/

# ls /opt/

cloudera cm-5.7.0

2、创建数据库

# mysql -uroot -p

Enter password: \\输入数据库密码

Welcome to the MariaDB monitor. Commands end with ; or \g.

Your MariaDB connection id is 10

Server version: 5.5.47-MariaDB MariaDB Server

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;