Ambari brief and Cluster Setup (on)

A, Ambari brief

1 .1 What is Ambari

Apache Ambari project aims to develop a configuration, management and monitoring software Apache Hadoop Hadoop cluster to simplify management. Ambari provides an intuitive, easy to use Hadoop management Web UI.

. 1 .2, Ambari function

It provides across any number of hosts Hadoop installation services step by step guide.
Processing Hadoop cluster service configuration.
It provides centralized management for the entire cluster to start, stop, and reconfigure the Hadoop services.
It provides a dashboard for monitoring the health and status Hadoop cluster.
Ambari indicators were collected using the index system.
Use Ambari Alert Framework alarm system and notifies you when you need your attention (for example, node fails, the lack of free disk space, etc.).

Second, prepare the environment

Note: the following three machines need to be

2 .1 virtual machine is ready

Cloning three virtual machines (node61, node62, node63), configure the corresponding host network IP, host name, turn off the firewall.

[root @ node62 ~] # systemctl Status firewalld # View firewall status
[root @ node62 ~] # systemctl Start firewalld # open the firewall service
[root @ node62 ~] # systemctl STOP firewalld # turn off the firewall service

[root @ node62 ~] # systemctl disable firewalld # Set the boot does not start

2.2, close SELINUX security module

Temporary closure: setenforce 0
modify the configuration file VI / etc / SELinux / config (restart commencement init 6): The SELINUX = enforcing SELINUX = disabled to
view the status SELINUX:
　. 1) / usr / sbin / -v sestatus
　　the SELinux Status: Enabled ( enabled: open; disabled: off)
　2) use the command: getenforce

2.3, install JDK

1) Create a module and software folders in the / opt directory of hadoop102

2) The jdk-8u144-linux-x64.tar.gz introduced into the hadoop102 the / opt / software directory

3) In the opt directory under the Linux system to see whether the import was successful package

4) Extract JDK to the / opt / module directory: [the root @ node61 Software] # the tar-8u181 -zxvf JDK-Linux-x64.tar.gz -C / opt / module /

5) Configure JDK environment variables

(1) to obtain the JDK path :: / opt / module / jdk1.8.0_181

(2) Open the / etc / profile file: vi / etc / profile

Add JDK path in the profile end of the file

#JAVA_HOME

export JAVA_HOME=/opt/module/jdk1.8.0_181

export PATH=$PATH:$JAVA_HOME/bin

(3) Save and exit:: wq

(4) file after the change to take effect: source / etc / profile

6) Test JDK is installed successfully: [root @ node61 ~] # java -version

7) The node61 the JDK and environment variables distributed to node62, node63 two hosts

[root@node61 ~]# scp -r /opt/module/ 192.168.18.162:/opt/module/

[root@node61 ~]# scp -r /opt/module/ 192.168.18.163:/opt/module/

[root@node61 ~]# scp -r /etc/profile 192.168.18.162:/etc/profile

[root@node61 ~]# scp -r /etc/profile 192.168.18.163:/etc/profile

2.4, SSH login avoid dense (recommended in advance)

Host configuration file: [root @ node61 ~] # vi / etc / hosts

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.18.161 node61
192.168.18.162 node62
192.168.18.163 node63

[root@node61 ~]# scp /etc/hosts 192.168.18.163:/etc/hosts
[root@node61 ~]# scp /etc/hosts 192.168.18.163:/etc/hosts

Configure node61 to node61, node62, node63 three hosts-free secret landing

1) generating public and private keys:

[root@node61 ~]# ssh-keygen -t rsa

Then knock (three carriage return), it will generate two files id_rsa (private), id_rsa.pub (public key)

2) the public key to be copied to the target machine the log dense Free

[root@node61 ~]# ssh-copy-id node61
[root@node61 ~]# ssh-copy-id node62
[root@node61 ~]# ssh-copy-id node63

Similarly the other two machines arranged densely Free login

2.5, modify yum aliyun source mirror (optional)

[root@node61 ~]# yum install -y wget

[root@node61 ~]# mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.bk

[root@node61 ~]# wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.rep

[root @ node61 ~] # yum makecache # The server package is downloaded to the local cache

2.6, installation ntp (CentOS7 Alternatively, CentOS6 required)

[root@node61 ~]# yum install -y ntp
[root@node61 ~]# chkconfig --list ntpd
[root@node61 ~]# chkconfig ntpd on
[root@node61 ~]# service ntpd start

2.7, shut down the Linux THP service (optional)

If you do not close transparent_hugepage, HDFS will be severely affected because of this performance.

Close transparent_hugepage method is:

[root@node61 ~]# vim /etc/grub.conf
添加 transparent_hugepage=never

[root@node61 ~]# vim /etc/rc.local
添加：
if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
echo never > /sys/kernel/mm/transparent_hugepage/defrag
fi
if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi
exit 0

After the restart, check with the following command:

[root@node61 yum.repos.d]#
cat /sys/kernel/mm/redhat_transparent_hugepage/enabled

always madvise [never]

There are [never] said THP is disabled

2.8, placed UMASK

The initial set user permissions to the directory created: [root @ node61 ~] # the umask 0022

2.9, prohibit offline update (optional)

Modify the file: vi /etc/yum/pluginconf.d/refresh-packagekit.conf

New: enabled = 0

Third, the installation Ambari cluster

Note: the following operations to the master node

3.1, create a local source

Production is due to the local source line installation Ambari too slow. Only you need to create a local source on the primary node.

3.1.1 Configuring HTTPD Service

HTTPD service system layer arranged to start automatically with the system

[root@node61 ~]# yum list installed | grep httpd
[root@node61 ~]# yum install -y httpd

[root@node61 ~]# systemctl status httpd.service

[root@node61 ~]# systemctl enable httpd.service

3.1.2 Installation Tools

Install a local source production tools

[root@node61 ~]# yum install yum-utils createrepo yum-plugin-priorities -y
[root@hadoop102 ~]# vi /etc/yum/pluginconf.d/priorities.conf

Add gpgcheck = 0 (off permission check)

3.1.3 download three tar decompression

Use wget to obtain, too slow, I do so, first with the Thunder to download locally and then upload it to the server.

Note about whether to download the full:

[root@node61 software]# tar -zxvf /opt/software/ambari-2.7.3.0-centos7.tar.gz -C /var/www/html/
[root@node61 software]# mkdir -p /var/www/html/hdp
[root@node61 software]# tar -zxvf /opt/software/HDP-3.1.0.0-centos7-rpm.tar.gz -C /var/www/html/hdp
[root@node61 software]# tar -zxvf /opt/software/HDP-UTILS-1.1.0.22-centos7.tar.gz -C /var/www/html/hdp

[root@node61 software]# tar -zxvf /opt/software/HDP-GPL-3.1.0.0-centos7-gpl.tar.gz -C /var/www/html/hdp

Note: the same user rights at all levels of the directory, not those who can not easily access ( [root @ node61 centos7] # chown root: root ./2.7.3.0-139/ -R )

3.1.4 Creating a local source

[root@node61 software]# cd /var/www/html/
[root@node61 html]# createrepo ./

3.1.5 download Ambari repository files to a directory on the host installation

[root@node61 html]# wget -nv http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.3.0/ambari.repo -O /etc/yum.repos.d/ambari.repo
2020-03-15 00:36:19 URL:http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.7.3.0/ambari.repo [376/376] -> "/etc/yum.repos.d/ambari.repo" [1]
[root@node61 html]#

3 .1.6 modify the configuration file, the configuration of the local source

Modify ambari.repo, configured as a local source (optionally, also modify /var/www/html/ambari/centos7/2.7.3.0-139/ambari.repo )

[root@node61 html]# vi /etc/yum.repos.d/ambari.repo
#VERSION_NUMBER=2.7.3.0-139
[ambari-2.7.3.0]
#json.url = http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json
name=ambari Version - ambari-2.7.3.0
baseurl=http://node61/ambari/centos7/2.7.3.0-139/
gpgcheck=0
gpgkey=http://node61/ambari/centos7/2.7.3.0-139/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

Modify hdp-util.repo, configured as a local source

[root@node61 html]# vci /var/www/html/hdp/HDP-UTILS/centos7/1.1.0.22/hdp-util.repo
[HDP-UTILS-1.1.0.22]
name=Hortonworks Data Platform Utils Version - HDP-UTILS-1.1.0.22
baseurl=http://node61/hdp/HDP-UTILS/centos7/1.1.0.22/
gpgcheck=0
enabled=1
priority=1

Modify hdp.repo, configured as a local source

[root@node61 html]# vi /var/www/html/hdp/HDP/centos7/3.1.0.0-78/hdp.repo

#VERSION_NUMBER=3.1.0.0-78
[HDP-3.1.0.0]
name=HDP Version - HDP-3.1.0.0
baseurl=http://node61/hdp/HDP/centos7/3.1.0.0-78/
gpgcheck=0
gpgkey=http://node61/hdp/HDP/centos7/3.1.0.0-78/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

[HDP-UTILS-1.1.0.22]
name=HDP-UTILS Version - HDP-UTILS-1.1.0.22
baseurl=http://node61/hdp/HDP-UTILS/centos7/1.1.0.22/
gpgcheck=0
gpgkey=http://node61/hdp/HDP-UTILS/centos7/1.1.0.22/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

Modify hdp.gpl.repo, configured as a local source

[root@node61 3.1.0.0-78]# vi /var/www/html/hdp/HDP-GPL/centos7/3.1.0.0-78/hdp.gpl.repo
#VERSION_NUMBER=3.1.0.0-78
[HDP-GPL-3.1.0.0]
name=HDP-GPL Version - HDP-GPL-3.1.0.0
baseurl=http://node61/hdp/HDP-GPL/centos7/3.1.0.0-78/
gpgcheck=0
gpgkey=http://node61/hdp/HDP-GPL/centos7/3.1.0.0-78/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

# Clear YUM cache: [root @ node61 ~] # yum Clean All
# The package information is downloaded to the local computer server together with the cache: [root @ node61 ~] # yum makecache
# See if there Ambari: [root @ node61 ~] # yum repolist

View Ambari the repository HDP resources.

You can also open a browser look:

3.2, Installing MySQL

Ambari use the default database is PostgreSQL, for storing metadata installed, you can use your own install the MySQL database as Ambari metadata database.

Note : must use the root user steps ; first uninstalled and then installed MySQL

Reference: https://lixinkuan.blog.csdn.net/article/details/99699411

3.3, install Ambari

1) Server-mounted ambari: [@ node61 the root ~] # yum the install-Server ambari

2) Copy driving mysql

Copy mysql-connector-java.jar to the / usr / share / java directory and renamed mysql-connector-java.jar

[root@node61 software]# mkdir /usr/share/java
[root@node61 software]# cp ./mysql-connector-java.jar /usr/share/java/

Copy mysql-connector-java.jar to / var / lib ambari-server at / / resources directory and renamed mysql-jdbc-driver.jar

[root@node61 software]# cp /usr/share/java/mysql-connector-java.jar /var/lib/ambari-server/resources/mysql-jdbc-driver.jar

Modify ambari.properties file

[root@node61 software]# vi /etc/ambari-server/conf/ambari.properties

添加：server.jdbc.driver.path=/usr/share/java/mysql-connector-java.jar

3.4, create a database in MySQL

1) Create a library ambari

[root @ node61 mysql-libs] # rpm -ivh MySQL-client-5.6.24-1.el6.x86_64.rpm
preparation ... ################## ############### [100%]
being upgraded / installed ...
. 1: the MySQL-Client-5.6.24-1.el6 ########### ###################### [100%]
[the root @ node61 MySQL-libs] #
[@ node61 MySQL the root-libs] #
[@ node61 the root MySQL- libs] # mysql -h 192.168.18.102 -P 3306 -u root -p123

mysql> create database ambari;

2) Use Ambari own script to create the table

mysql> use ambari;
Database changed
mysql> source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql;
Query OK, 0 rows affected (0.01 sec)

Query OK, 0 rows affected (0.00 sec)

Query OK, 0 rows affected (0.00 sec)

3.5, configuration Ambari

Execution: [@ node61 the root ~] # ambari-Server Setup

Below is configured to perform the process, follow the prompts

1) prompts whether custom settings. Input: y

Customize user account for ambari-server daemon [y/n] (n)? y

2) ambari-server account.

Enter user account for ambari-server daemon (root):

If you directly enter the root user is the default selection

3) Set JDK. Input: 2

Checking JDK...
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Custom JDK
==============================================================================
Enter choice (1): 2

If the above 3 selected custom JDK, you need to set the JAVA_HOME. Input: /opt/module/jdk1.8.0_144

WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.

WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.

Path to JAVA_HOME: /opt/module/jdk1.8.0_18

Validating JDK on Ambari Server...done.

Completing setup...

4) database configuration. Select: y

Configuring database...

Enter advanced database configuration [y / n] (n)? Y

5) Select the type of database. Input: 3

Configuring database...
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
==============================================================================
Enter choice (1): 3

6) The specific configuration settings in a database, according to the actual input, and if the same brackets, can directly enter. If you want to rename, enter.

Hostname (localhost): 192.168.18.102
Port (3306):
Database name (ambari):
Username (ambari): root
Enter Database Password (bigdata):
Re-enter password:
Configuring ambari database...

7) The Ambari database scripts into database

WARNING: Before starting Ambari Server, you must run the following DDL directly from the database shell to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
Proceed with configuring remote database connection properties [y/n] (y)? y
Extracting system views...
ambari-admin-2.7.3.0.139.jar
....
Ambari repo file contains latest json url http://public-repo-1.hortonworks.com/HDP/hdp_urlinfo.json, updating stacks repoinfos with it...
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.

If you use your own defined database, you must import Ambari the sql script before starting the Ambari service.