Ambari 与hdp 集群搭建教程

Ambari 与hdp 集群搭建教程

集群环境os为centos6.8 server

1,各个节点进行：网络（静态IP）配制（

192.168.10.30	master	8g
192.168.10.31	slave1	8g
192.168.10.32	Slave2	8g
192.168.10.33	Slave3	8g
192.168.10.34	Slave4	8g
192.168.10.35	Slave5	8g

Vi /etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE=eth0

HWADDR=00:50:56:A7:45:10

TYPE=Ethernet

UUID=010e7a3c-f81c-4914-a69a-01da66b9cfb3

ONBOOT=yes

NM_CONTROLLED=yes

BOOTPROTO=static

IPADDR=192.168.10.30

NETMASK=255.255.240.0

GATEWAY=192.168.2.1

DNS1=202.98.96.68

DNS2=202.98.96.69

Vi /etc/sysconfig/network

NETWORKING=yes

HOSTNAME=master

GATEWAY=192.168.2.1

重启网络服务：

/etc/init.d/network restart

2,Hosts配置

在各个节点：vi /etc/hosts

添加集群主机：

192.168.10.30 master

192.168.10.31 slave1

192.168.10.32 slave2

192.168.10.33 slave3

192.168.10.34 slave4

192.168.10.35 slave5

………………………….

3,关闭防火墙和SELinux（各个节点）。

chkconfig iptables off （重启后生效）

vi /etc/selinux/config

修改SELINUX=disabled （重启后永久生效

Ssh 无密码登录

在master 节点下：

ssh-keygen -t rsa

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

chmod 600 ~/.ssh/authorized_keys

scp ~/.ssh/authorized_keys root@slave1:~/.ssh/

scp ~/.ssh/authorized_keys root@slave2:~/.ssh/

……………………..

4,Ntp 即时服务

以master 为ntp校准服务器

Master 的 vi /etc/ntp.conf

restrict 127.0.0.1

restrict -6 ::1

# Hosts on local network are less restricted.

#restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap

restrict 192.168.10.0 mask 255.255.240.0 nomodify notrap //允许的网络字段连接ip限制

server 127.127.1.0 perfer //以本机时间为准

fudge 127.127.1.0 stratum 10

#server 0.centos.pool.ntp.org iburst //全部注释掉

#server 1.centos.pool.ntp.org iburst

#server 2.centos.pool.ntp.org iburst

#server 3.centos.pool.ntp.org iburst

各个客户端(slave1,2,3,4,5)配置如下：

Vi /etc/ntp.conf

Server master //指定ntp 服务器,为master

#server 0.centos.pool.ntp.org iburst //全部注释掉

#server 1.centos.pool.ntp.org iburst

#server 2.centos.pool.ntp.org iburst

#server 3.centos.pool.ntp.org iburst

启动全部节点ntp服务；

service ntpd start

检查是否成功，用ntpstat命令查看同步状态，出现以下状态代表启动成功

请求服务器前，请先使用ntpdate手动同步一下时间：ntpdate -umaster (主节点ntp服务器)

这里可能出现同步失败的情况，请不要着急，一般是本地的ntp服务器还没有正常启动，一般需要等待5-10分钟才可以正常同步。启动服务：

service ntpd start

5,在各个节点jdk配置,：

rpm –ivh jdk-8u101-linux-x64.rpm

此时jdk安装目录为/usr/java/….

配置jdk环境：

Vi /etc/profile

export JAVA_HOME=/usr/java/jdk1.8.0_101

export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

export PATH=$PATH:$HOME/bin:$JAVA_HOME/bin

source /etc/profile

6,安装http服务器(主节点)

yum install httpd

service httpd start

#修改网站默认的跟目录

vi /etc/httpd/conf/httpd.conf

# DocumentRoot: Thedirectory out of which you will serve your

# documents. Bydefault, all requests are taken from this directory, but

# symbolic links andaliases may be used to point to other locations.

DocumentRoot"/var/www/html"

# This should bechanged to whatever you set DocumentRoot to.

<Directory"/var/www/html">

7,安装mysql(主节点)

rpm –ivh MySQL-server-5.5.36-1.el6.x86_64.rpm

rpm –ivh MySQL-client-5.5.36-1.el6.x86_64.rpm

service mysql start

service mysql start

4.mysqladmin -u root password 'xxxx'

5.mysql -uroot -pxxxx (进入mysql命令行)

进入mysql命令行之后，开始执行下面sql语

#创建用户并分配权限

CREATE USER 'ambari'@'master'IDENTIFIED BY 'ambari';
GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'master';

GRANT ALL PRIVILEGES ON *.* TO root@'master';

FLUSH PRIVILEGES;

8,创建用户组及用户（在主节点）

Groupadd ambari

添加ambari安装、运行用户到组.

useradd -G ambari ambari

9, 配置安装源

注:鉴于多数集群生产环境不能上网,所以安装时采用本地源的方式.具体为先下载附录中的ambari、HDP、HDP-UTILS的tar包然后再配置到本地源中。

#ambari和HDP(含HDP-UTILS)源

a)Ambari2.2.2源

操作系统	格式	URL地址
RedHat 6 CentOS 6 Oracle Linux 6	Base URL	http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.2.2.0
	Repo File	http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.2.2.0/ambari.repo
	Tarball md5 \| asc	http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.2.2.0/ ambari-2.2.2.0-centos6.tar.gz

b)HDP 2.4源

操作系统	版本号	源名称	格式	URL地址
RedHat 6 CentOS 6 Oracle Linux 6	HDP-2.4.2.0	HDP	Base URL	http://public-repo-1.hortonworks.com/HDP/centos6/2.x/ updates/2.4.2.0
			Repo File	http://public-repo-1.hortonworks.com/HDP/centos6/2.x/ updates/2.4.2.0/hdp.repo
			Tarball md5 \| asc	http://public-repo-1.hortonworks.com/HDP/centos6/2.x/ updates/2.4.2.0/HDP-2.4.2.0-centos6-rpm.tar.gz
		HDP-UTILS	Base URL	http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/ repos/centos6
		HDP-UTILS	Repo File	http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/ repos/centos6/HDP-UTILS-1.1.0.20-centos6.tar.gz

1) 下载如下组件并拷贝到主节点的httpd网站根目录,即/var/www/html内后解压

Ambari 2.2.2:

http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.2.2.0/ambari-2.2.2.0-centos6.tar.gz

HDP 2.4.2:

http://s3.amazonaws.com/public-repo-1.hortonworks.com/HDP/centos6/2.x/2.4-latest/HDP-2.4-latest-centos6-rpm.tar.gz

HDP-UTILS-1.1.0.20：

http://s3.amazonaws.com/public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/repos/centos6/HDP-UTILS-1.1.0.20-centos6.tar.gz

2) 配置ambari、HDP、HDP-UTILS的源

注:事实证明HDP、HDP-UTILS源不需要配置，ambari安装时可以指定本地源,并在各个节点里创建repo文件.ambari的源需要配置，因为安装采用yum install ambari方式.

vi ambari.repo(在http目录下，即解压目录：/var/www/html/)

#VERSION_NUMBER=2.2.2.0-460

[Updates-ambari-2.2.2.0]

name=ambari-2.2.2.0 -Updates

baseurl=http://192.168.10.30/AMBARI-2.2.2.0/centos6/2.2.2.0-460/

gpgcheck=1

gpgkey=http://192.168.10.30/AMBARI-2.2.2.0/centos6/2.2.2.0-460/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins

enabled=1

priority=1

vi hdp.repo(在http目录下，即解压目录：/var/www/html/)

#VERSION_NUMBER=2.4.2.0-258

[HDP-2.4.2.0]

name=HDP Version -HDP-2.4.2.0

baseurl=http://192.168.10.30/HDP/centos6/2.x/updates/2.4.2.0

gpgcheck=1

gpgkey=http://192.168.10.30/HDP/centos6/2.x/updates/2.4.2.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins

enabled=1

priority=1

[HDP-UTILS-1.1.0.20]

name=HDP Utils Version- HDP-UTILS-1.1.0.20

baseurl=http://192.168.10.30/HDP-UTILS-1.1.0.20/repos/centos6

gpgcheck=1

gpgkey=http://192.168.10.30/HDP/centos6/2.x/updates/2.4.2.0/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins

enabled=1

priority=1

将ambari.repo ,hdp.repo复制到/etc/ yum.repos.d/目录下

接着执行以下指令：

yum clean all

yum list update

yum makecache

yum repolist

10. 执行安装

Yum方式安装ambari。

安装ambari2.2.2

1）安装ambari

yum install ambari-server

2）配置amabri

ambari-server setup
Customize user account for ambari-server daemon [y/n] (n)?
输入：y
Enter user account for ambari-server daemon (root):

注:这里亦可选择用户amabri,届时会让提供该用户的密码等信息.
输入：root

检查防火墙是否关闭
Adjusting ambari-server permissions and ownership...
Checking firewall...
WARNING: iptables is running. Confirm the necessary Ambari ports areaccessible. Refer to the Ambari documentation for more details on ports.

OK to continue [y/n] (y)?
输入：y

检查JDK
Checking JDK...

[1] - Oracle JDK 1.7 + Java Cryptography Extension (JCE)Policy Files 7
[2] - Oracle JDK 1.6 + Java Cryptography Extension (JCE) Policy Files 6
[3] - Custom JDK
==============================================================================
Enter choice (1):
输入：3
输入：/自己的jdk安装目录

Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL
[4] - PostgreSQL
==============================================================================
Enter choice (1):
输入：3

Hostname (localhost):
Port (3306):
Database Name (ambari):
Username (ambari):
输入：（什么也不输入，直接“回车”）

Enter Database Password (cluster):
Re-enter password:
输入：ambari

WARNING: Before starting Ambari Server, youmust run the following DDL against the database to create the schema:/var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
注:这里需要以mysql的amabri用户在ambari数据库里source上面的sql文件，作用时初始化ambari数据库。