A, ambari architecture parsing
Second, the basic configuration of the environment
In the two nodes as an example to assembly Hadoop distributed cluster, employed herein system version is Centos7 1511, as shown in the following table:
| CPU name |
RAM |
hard disk |
IP addresses |
Roles |
| master |
8192MB |
100G |
192.168.200.6 |
Ambari-Server Ambari-Agent |
| slaves |
8192MB |
100G |
192.168.200.7 |
Ambari-Agent |
Each version of the software needed to build information
① ambari-2710-centos7targz
②jdk-8u151-linux-x64.tar.gz
③ HDP-3010-centos7-rpmtargz
④ HDP-UTILS-1.1.0.22-centos7.tar
⑤CentOS-7-x86_64-DVD-1511.iso
1. In iaas create two cloud hosting platform
- Host name, host name resolution configuration
vi /etc/hosts
- Mirror mount and configure local yum source
First, the desired image, the installation package, etc., by secureFX uploaded to the master 's / opt directory
Master node
3.1 into the / opt directory, extract ambari.tar.gz archive
[root@master ~]# cd /opt
[root@master opt]# ls
ambari-2.7.1.0-centos7.tar.gz HDP-3.0.1.0-centos7-rpm.tar.gz
hundreds HDP-useful-1.1.0.22-centos7.tar
[root@master opt]# tar -zxvf ambari-2.7.1.0-centos7.tar.gz
After the extraction is completed to obtain a ambari folder
[root@master ~]# ls
CentOS-7-x86_64-DVD-1511.iso
[root@master ~]# mount -o loop CentOS-7-x86_64-DVD-1511.iso /mnt/
[root@master ~]# mkdir /opt/centos
[root@master ~]# cp -rvf /mnt/* /opt/centos/
[root@master ~]# cd /etc/yum.repos.d/
[root@master yum.repos.d]# ls
CentOS-Base.repo CentOS-Debuginfo.repo CentOS-Sources.repo
CentOS-CR.repo CentOS-fasttrack.repo CentOS-Vault.repo
[root@master yum.repos.d]# rm -rf *
[root@master yum.repos.d]# vi local.repo
[Centos]
name=centos
baseurl = file: /// opt / CentOS
gpgcheck=0
enable=1
[Ambari]
name=ambari
baseurl = file: ///opt/ambari/centos7/2.7.1.0-169/
gpgcheck=0
enable=1
Save and exit
Clean up the source, and the source list
[root@master ~]# yum clean all
Loaded plugins: fastestmirror
Cleaning repos: ambari centos
Cleaning up everything
Cleaning up list of fastest mirrors
[root@master ~]# yum list
[root@master ~]# yum install vsftpd
[root@master ~]# vi /etc/vsftpd/vsftpd.conf
Add to
anon_root=/opt/
3.2 Configuring boot vsftpd
[root@master ~]# systemctl enable vsftpd
[root@master ~]# systemctl start vsftpd
Slaver node
[root@slaver ~]# cd /etc/yum.repos.d/
[root@slaver yum.repos.d]# ls
CentOS-Base.repo CentOS-Debuginfo.repo CentOS-Sources.repo
CentOS-CR.repo CentOS-fasttrack.repo CentOS-Vault.repo
[root@slaver yum.repos.d]# rm -rf *
[root@slaver yum.repos.d]# vi ftp.repo
[Centos]
name=centos
baseurl = ftp: //192.168.200.6/centos
gpgcheck=0
enable=1
[Ambari]
name=ambari
baseurl = ftp: //192.168.200.6/ambari/centos7/2.7.1.0-169/
gpgcheck=0
enable=1
Save and exit
- Configuring NTP Service
Master node
[root@master ~]# yum -y install ntp
[root@master ~]# vi /etc/ntp.conf
4.1 The original server comment
Add to
server 127.127.1.0
fudge 127.127.1.0 stratum 10
[root@master ~]# systemctl enable ntpd
[root@master ~]# systemctl start ntpd
Slaver node
[root@slaver ~]# yum -y install ntpdate
[root@slaver ~]# ntpdate master
[root@slaver ~]# systemctl enable ntpdate
- Configuring SSH -free access secret
Master node
[root@master ~]# yum install openssh-clients
[root@master ~]# ssh-keygen
[root@master ~]# ssh-copy-id slaver.hadoop
Slaver node
[root@slaver ~]# yum install openssh-clients
[root@slaver ~]# ssh-keygen
[root@slaver ~]# ssh-copy-id master.hadoop
- Disabling Transparent Huge Pages
Operating system background have called khugepaged process, it would have been scanning all process memory, where possible circumstances would 4kpage exchanged for Huge Pages and the , in the process, for a variety of memory allocation operations are required for each activity kinds of memory locks, direct memory access affect the performance of the program, and this process is transparent to the application, can not be controlled at the application level , for specifically for 4k page optimization program, it may cause performance degradation random phenomenon.
master & slaver
[root@master ~]# echo never > /sys/kernel/mm/transparent_hugepage/enabled
[root@master ~]# echo never > /sys/kernel/mm/transparent_hugepage/defrag
[root@master ~]# cat /sys/kernel/mm/transparent_hugepage/enabled
always advise [never]
After the restart fails, we need to perform again
- Installation configuration jdk
Master node
[root@master ~]# mkdir /usr/jdk64/
[root@master ~]# tar -zxvf jdk-8u151-linux-x64.tar.gz -C /usr/jdk64/
[root@master ~]# vi /etc/profile
7.1 Add the following two variables
export JAVA_HOME=/usr/jdk64/jdk1.8.0_151
export PATH=$JAVA_HOME/bin:$PATH
[root@master ~]# source /etc/profile
[root@master ~]# java -version
java version "1.8.0_151"
Java(TM) SE Runtime Environment (build 1.8.0_151-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)
Slaver node
[root@slaver ~]# mkdir /usr/jdk64/
[root@slaver ~]# scp 192.168.200.6://root/jdk-8u151-linux-x64.tar.gz .
jdk-8u151-linux-x64.tar.gz 100% 181MB 90.5MB/s 00:02
[root@slaver ~]# tar -zxvf jdk-8u151-linux-x64.tar.gz -C /usr/jdk64/
[Slaves root @ ~] # vi / etc / profile
export JAVA_HOME=/usr/jdk64/jdk1.8.0_151
export PATH=$JAVA_HOME/bin:$PATH
[root@slaver ~]# source /etc/profile
[root@slaver ~]# java -version
java version "1.8.0_151"
Java(TM) SE Runtime Environment (build 1.8.0_151-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)
- Install httpd service
8.1 install httpd
[root@master ~]# yum -y install httpd
[root@master ~]# systemctl enable httpd.service
[root@master ~]# systemctl start httpd.service
[root@master ~]# systemctl status httpd.service
8.2 decompression HDP
The / opt directory
HDP-3.0.1.0-centos7-rpm.tar.gz
HDP-utils-1.1.0.22-centos7.tar
Decompressed and copied to / var / www / html
[root@master opt]# tar -zvxf HDP-3.0.1.0-centos7-rpm.tar.gz -C /var/www/html/
[Root @ master opt] # tar -zvxf HDP-UTILS-1.1.0.22-centos7.tar -C / var / www / html /
Three, ambari-server installation configuration
Master node
[root@master ~]# yum -y install ambari-server
1. install MariaDB database
[root@master ~]# yum install mariadb mariadb-server mysql-connector-java
1.1 Start the database service
[root@master ~]# systemctl enable mariadb
[root@master ~]# systemctl start mariadb
1.2 Configuration Database Service
[root@master ~]# mysql_secure_installation
Root password the SET? [The Y-/ the n-] the y- (set a password for BigData )
Remove anonymous users? [Y / n] and
Disallow root login remotely? [Y/n] n
Remove test database and access to it? [Y/n] y
Reload privilege tables now? [Y / n] and
[root@master ~]# mysql -uroot -pbigdata
MariaDB [(none)]> create database ambari;
MariaDB [(none)]> grant all privileges on ambari.* to 'ambari'@'localhost' identified by 'bigdata';
MariaDB [(none)]> grant all privileges on ambari.* to 'ambari'@'%' identified by 'bigdata';
MariaDB [(none)]> use ambari;
MariaDB [ambari]> source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
MariaDB [ambari]> quit
- Configuring ambari-server
[root@master ~]# vi /etc/profile
Add to
export buildNumber=2.7.1.0
[root@master ~]# ambari-server setup
Using python /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'disabled'
Customize user account for ambari-server daemon [y/n] (n)? n
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Custom JDK
==============================================================================
Enter choice (1): 2
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: /usr/jdk64/jdk1.8.0_151
Validating JDK on Ambari Server...done.
Check JDK version for Ambari Server...
JDK version found: 8
Minimum JDK version is 8 for Ambari. Skipping to setup different JDK for Ambari Server.
Checking GPL software agreement...
GPL License for LZO: https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html
Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)? n
Completing setup...
Configuring database...
Enter advanced database configuration [y / n] (n)? and
Configuring database...
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
==============================================================================
Enter choice (1): 3
Hostname (localhost):
Port (3306):
Database name (ambari):
Username (ambari):
Enter Database Password (bigdata):
Configuring ambari database...
Should ambari use existing default jdbc /usr/share/java/mysql-connector-java.jar [y/n] (y)?
Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
Proceed with configuring remote database connection properties [y/n] (y)?
Extracting system views...
sambari-admin-2.7.1.0.169.jar
....
Ambari repo file doesn't contain latest json url, skipping repoinfos modification
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.
[root@master ~]# ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar
- Start ambari-server service
[root@master ~]# ambari-server start
DB configs consistency check: no errors and warnings were found.
Ambari Server 'start' completed successfully.
Four, ambari-agent Installation Configuration
master & slaver
1. Installation ambari-agent
[root@master ~]# yum -y install ambari-agent
[root@slaver ~]# yum -y install ambari-agent
- Modify variables
[root@master ~]# vi /etc/ambari-agent/conf/ambari-agent.ini
[Slaves root @ ~] # we /etc/ambari-agent/conf/ambari-agent.ini
The [server] modified as master.hadoop
hostname=master.hadoop
3. Restart ambari-agent service
[root@master ~]# ambari-agent restart
[root@slaver ~]# ambari-agent restart
V. deploy Hadoop clusters
1. Access WEB interface
Open your browser and enter the IP: 8080 visit ambari the web interface
This machine is http://192.168.200.6:8080
The default user name admin
The default password is admin
- Creating a new cluster
3.1 Naming cluster
3.2 Select local warehouse sources
Select redhat 7
http://192.168.200.6/HDP/centos7/3.0.1.0-187/
http://192.168.200.6/HDP-UTILS/centos7/1.1.0.22/
3.3 Configuring the host name
3.4 host state registration check
3.5 Service installation options
Here we have chosen
YARN + MapReduce2
HDFS
ZooKeeper
Ambari Metrics
Spark2
3.6 Service assigned master
Default.
3.7 Node Service Selection
Set Grafana password
Next to Default
Confirm deployment
Begin deployment