1. Preparing for installation of clouderaManager environment

Prepare three virtual machines, one of which will be our master node and install our ClouderaManager Server and ClouderaManager agent, and the other two will be our slave nodes and only our clouderaManager agent will be installed

The machine plan is as follows

Server IP	192.168.43.110	192.168.43.120	192.168.43.130
CPU name	cm1	cm2	cm3
Firewall	shut down	shut down	shut down
selinux	shut down	shut down	shut down
jdk	installation	installation	installation
SSH password-free login	Yes	Yes	Yes
mysql database	Yes	no	no
Server memory	20G	16G	16G
Master-slave node	Master, install CM	From	From
Server Audit	4	4	4

. 1 .1 , building server 3

Switch root user

su root

Configure permissions

vim /etc/sudoers

1 .2 , modify the network card

vim /etc/udev/rules.d/70-persistent-net.rules

. 1 .3 , ip address modification

vim /etc/sysconfig/network-scripts/ifcfg-eth0

1 .4 , change the hostname

vim /etc/sysconfig/network

1 .5 , change the host name and IP address mapping

vim /etc/hosts

. 1 .6 , turn off the firewall

service iptables stop
chkconfig iptables off

. 1 .7 , closed selinux

vim /etc/selinux/config
SELINUX=disabled

1.8 , SSH password-free login

Operate on three servers

Generate public and private keys:

ssh-keygen -t rsa

然后敲（三个回车），就会生成两个文件id_rsa（私钥）、id_rsa.pub（公钥）

Copy the public key to the target machine to log in without secret

ssh-copy-id cm1
ssh-copy-id cm2
ssh-copy-id cm3

. 1 .9 , installation MySQL

Install MySQL and its driver

Install mysql online through yum source

yum  install  mysql  mysql-server  mysql-devel
/etc/init.d/mysqld start
/usr/bin/mysql_secure_installation

Enter the mysql client and execute the following command

grant all privileges on *.* to 'root'@'%' identified by '123456' with grant option;
flush privileges;

The mysql-connector-java-5.1.27- bin.jar copied to / usr / Share / Java of the path, and rename

mv mysql-connector-java-5.1.27-bin.jar mysql-connector-java.jar

Distribute the driver to other nodes

. 1 .10 , installation JDK

1) Create /usr/java directory in cm1

mkdir /usr/java

2) Upload jdk-8u144-linux-x64.tar.gz to cm1, and unzip it to the /usr/java directory.

tar -zxvf jdk-8u144-linux-x64.tar.gz -C /usr/java/

3) Configure the JAVA_HOME environment variable

(1) Open the /etc/profile file

vi /etc/profile

Add the JDK path at the end of the profile file

#JAVA_HOME
export JAVA_HOME=/usr/java/jdk1.8.0_144
export PATH=$PATH:$JAVA_HOME/bin

(2) Let the modified file take effect

source /etc/profile

4) Test whether the JDK is successfully installed

java -version

5) Distribute the JDK and environment variables in cm1 to cm2 and cm3 hosts

scp -r /usr/java/ root@cm2:/usr/
scp -r /etc/profile root@cm2:/etc/

Source on cm2 and cm3 respectively

. 1 .11 , three machine clock synchronization

Three machines need to perform clock synchronization operations to ensure that the three machines have the same time

Start NTP service service ntpd start

crontab –e

*/1 * * * * /usr/sbin/ntpdate ntp4.aliyun.com;

1 . 12 , the lifting of linux system to open the file the largest number of restrictions

All three machines need to be executed

vi /etc/security/limits.conf

Add the following

* soft noproc 11000

* hard noproc 11000

* soft nofile 65535

* hard nofile 65535

. 1 . 13 is , provided linux swap memory

All three machines must be executed

Excuting an order

echo 10 > /proc/sys/vm/swappiness

And edit the file sysctl.conf:

vim /etc/sysctl.conf

Add or modify

vm.swappiness = 0

All three machines must execute:

echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled

And edit the file rc.local:
vim /etc/rc.local

echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled

Two, clouderaManager installation resource download

Step 1 : Download the installation resources and upload them to the server

We install the CM5.14.0 version here, we need to download the following resources, a total of four files can be

Download the compressed package of cm5

Download link: http://archive.cloudera.com/cm5/cm/5/

Specific file address:

http://archive.cloudera.com/cm5/cm/5/cloudera-manager-el6-cm5.14.0_x86_64.tar.gz

Download cm5 of parcel package

download link:

http://archive.cloudera.com/cdh5/parcels/

The specific download address of the first file:

http://archive.cloudera.com/cdh5/parcels/5.14.0/CDH-5.14.0-1.cdh5.14.0.p0.24-el6.parcel

The specific download address of the second file:

http://archive.cloudera.com/cdh5/parcels/5.14.0/CDH-5.14.0-1.cdh5.14.0.p0.24-el6.parcel.sha1

The specific download address of the third file:

http://archive.cloudera.com/cdh5/parcels/5.14.0/manifest.json

Upload these four installation packages to the /export/softwares path of the first machine

Step 2 : Unzip the compressed package to the specified path

Unzip the CM installation package to the /opt path

tar -zxvf cloudera-manager-el6-cm5.14.0_x86_64.tar.gz -C /opt/

Step 3 : Copy the three files of our parcel package to the corresponding path

Our parcel contains three files, copy them to the /opt/cloudera/parcel-repo path, and remember that there is a file that needs to be renamed

cp CDH-5.14.0-1.cdh5.14.0.p0.24-el6.parcel CDH-5.14.0-1.cdh5.14.0.p0.24-el6.parcel.sha1 manifest.json  /opt/cloudera/parcel-repo/

Rename CDH-5.14.0-1.cdh5.14.0.p0.24-el6.parcel CDH-5.14.0-1.cdh5.14.0.p0.24-el6.parcel.sha1

cd /opt/cloudera/parcel-repo/
mv CDH-5.14.0-1.cdh5.14.0.p0.24-el6.parcel.sha1 CDH-5.14.0-1.cdh5.14.0.p0.24-el6.parcel.sha

Step 4 : Add ordinary users to all nodes and give sudo permissions

Add ordinary users on the cm1 machine and give sudo permissions

Execute the following command to create a normal user cloudera-scm

useradd --system --home=/opt/cm-5.14.0/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm

Grant sudo permissions to ordinary users of cloudera-scm

visudo

cloudera-scm ALL=(ALL) NOPASSWD: ALL

Step 5 : Change the configuration file of the master node

Change the configuration file on the cm1 machine

vim /opt/cm-5.14.0/etc/cloudera-scm-agent/config.ini

server_host=cm1

Step 6 : Distribute the installation package in the /opt directory to other machines

Distribute the installation package of the first machine to other machines

cd /opt
scp -r cloudera/ cm-5.14.0/ cm2:/opt

Step 7 : Create some database standby

Create a database on the cm1 machine

hive database

create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;

Cluster monitoring database

create database amon DEFAULT CHARSET utf8 COLLATE utf8_general_ci;

hue database

create database hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci;

oozie database

create database oozie DEFAULT CHARSET utf8 COLLATE utf8_general_ci;

Corresponding database authorization

grant all on amon.* to 'reports'@'%' identified by '123456';
grant all on oozie.* to 'oozie'@'%' identified by '123456';
grant all on hue.* to 'hue'@'%' identified by '123456';
grant all on hive.* to 'hive'@'%' identified by '123456';
grant all on *.* to 'root'@'%' identified by '123456';

Step 8 : Prepare the driver package for database connection

Prepare a database connection driver jar package on all machines and place it under the /usr/share/java path

Prepare a mysql driver connection package and put it in the /usr/share/java path

wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.45.tar.gz
tar -zxvf mysql-connector-java-5.1.45.tar.gz
cd /export/softwares/mysql-connector-java-5.1.45
cp mysql-connector-java-5.1.45-bin.jar /usr/share/java/mysql-connector-java.jar

Copy the driver package to other machines

cd /usr/share/java
scp mysql-connector-java.jar cm2:$PWD

Step 9 : Create a database for clouderaManager

node01 server execution

/opt/cm-5.14.0/share/cmf/schema/scm_prepare_database.sh mysql -hcm1  -uroot -p123456 --scm-host cm1 scm root 123456

Command description: / opt / cm - 5.14 . 0 /share/cmf/schema/scm_prepare_database. sh database type -h database host- u database user name- p database password-scm-host cm host database name user name password

Step 10 : Start the service

The master node starts clouderaManager Server and ClouderaManager agent services

/opt/cm-5.14.0/etc/init.d/cloudera-scm-server start
/opt/cm-5.14.0/etc/init.d/cloudera-scm-agent start

Start ClouderaManager agent service from nodes cm2 and cm3

/opt/cm-5.14.0/etc/init.d/cloudera-scm-agent start

The eleventh step : browser page access

http://cm1:7180/cmf/login

Default username admin

Password admin

Lift the limit on the maximum number of open files in the Linux system