The following steps are valid and correct! ! ! ! !
1. Basic environment preparation
1) Software list 3
Centos7 (64-bit) servers, the address is as follows:
jdk1.8
Mysql5.7 (version 5.7 is required!!)
mysql-connector-java-5.1.42-bin.jar
cloudera-manager-centos7- cm5.8.5_x86_64.tar.gz (http://archive.cloudera.com/cm5/cm/5)
CDH-5.8.5-1.cdh5.8.5.p0.5-el7.parcel(http://archive .cloudera.com/cdh5/parcels/5.8.5/)
CDH-5.8.5-1.cdh5.8.5.p0.5-el7.parcel.sha1(http://archive.cloudera.com/cdh5/parcels/ 5.8.5/)
manifest.json (http://archive.cloudera.com/cdh5/parcels/5.8.5/)
2) Server address
192.168.56.1 (this address maps your own server address)
192.168.56.2 (this address maps your own server address)
192.168.56.3 (this address maps your own server address)
3) Modify hostname
vim /etc/hostname
After modifying the host name cdh*, do the following
service network restart
4) Configure hosts
vim /etc/hosts
192.168.56.1 cdh1
192.168.56.2 cdh2
192.168.56.3 cdh3
5) Secret-free access
ssh-keygen -t rsa
//三台服务器全都执行以下命令
ssh-copy-id root@cdh1
ssh-copy-id root@cdh2
ssh-copy-id root@cdh1
6) Turn off the firewall
systemctl stop firewalld
systemctl disable firewalld
7) Close SELINUX
setenforce 0
vim /etc/sysconfig/selinux
SELINUX=disabled
8) Configure NTP
yum -y install ntp
systemctl start ntpd
systemctl enable ntpd
在cdh1节点
vim /etc/ntp.conf
server 0.cn.pool.ntp.org
server 1.cn.pool.ntp.org
server 2.cn.pool.ntp.org
server 3.cn.pool.ntp.org
restrict 0.cn.pool.ntp.org nomodify notrap noquery
restrict 1.cn.pool.ntp.org nomodify notrap noquery
restrict 2.cn.pool.ntp.org nomodify notrap noquery
restrict 3.cn.pool.ntp.org nomodify notrap noquery
server 127.127.1.0
fudge 127.127.1.0 stratum 10
systemctl start ntpd.service
systemctl enable ntpd.service
ntpstat
在cdh[2-3]节点
vim /etc/ntp.conf
server cdh1 prefer
systemctl start ntpd.service
systemctl enable ntpd.service
ntpdate -u cdh1
ntpstat
9) Install JDK
mkdir /usr/local/java
cp /home/ jdk-8u131-linux-x64.gz /usr/local/java/
cd /usr/local/java/
tar xvf jdk-8u131-linux-x64.gz
rm jdk-8u131-linux-x64.gz
scp -r /usr/local/java/ root@cdh2:/usr/local/
scp -r /usr/local/java/ root@cdh3:/usr/local/
Set environment variables
vi /etc/profile
export JAVA_HOME=/usr/local/jdk
export JRE_HOME=$JAVA_HOME/jre
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$PATH
java -version
10) Install MySQL
Delete your own database
rpm -qa | grep mariadb
rpm -e –nodeps 包名
rm -rf /etc/my.cnf
Download dependencies and create users and groups
yum -y install perl perl-devel autoconf
groupadd mysql
useradd -g mysql mysql
id mysql
Please refer to the following link for installation:
https://blog.csdn.net/baidu_32872293/article/details/80557668
Set the database to allow remote login
mysql -u root -p
use mysql;
update user set host=’%’ where user=’root’;
If the error is reported, it may have been set up, use the following statement to view
select host from user where user = ‘root’;
flush privileges;
Change the database character driver
set character_set_client=utf8;
set character_set_connection=utf8;
set character_set_results=utf8;
set character_set_server=utf8;
set character_set_system=utf8;
View modification results
show variables like ‘character_set_%’;
Close the database security mode, so that the database can be added, deleted, and modified
SET SQL_SAFE_UPDATES=0;
Create a database and make the local root user have access
mysql -u root -p
create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database am DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database oozie DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
grant all privileges on *.* to ‘root’@’cdh1’ identified by ‘000000’ with grant option;
flush privileges;
Install CDH
cp /home/cloudera-manager-centos7-cm5.8.5_x86_64.tar.gz /opt/
cd /opt/
tar xvf cloudera-manager-centos7-cm5.8.5_x86_64.tar.gz
rm cloudera-manager-centos7-cm5.8.5_x86_64.tar.gz
Modify agent configuration
vim /opt/cm-5.8.5/etc/cloudera-scm-agent/config.ini
server_host=cdh1
scp -r /opt/cm-5.8.5 root@cdh2:/opt/
scp -r /opt/cm-5.8.5 root@cdh3:/opt/
Create scm management user for all nodes
useradd --system --home=/opt/cm-5.8.5/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm
cdh1 node
cp /home/ mysql-connector-java-5.1.47-bin /opt/cm-5.8.5/share/cmf/lib/
Initialization script configuration database
/opt/cm-5.8.5/share/cmf/schema/scm_prepare_database.sh mysql -h cdh1 -uroot -p000000 --scm-host cdh1 scm scm scm
Configure permissions for the scm database
mysql -u root -p
grant all on *.* to ‘scm’@’%’ identified by ‘scm’ with grant option;
flush privileges;
exit;
Copy the installation package file to the default installation path
cp /home/CDH-5.8.5-1.cdh5.8.5.p0.5-el7.parcel /opt/cloudera/parcel-repo/
cp /home/CDH-5.8.5-1.cdh5.8.5.p0.5-el7.parcel.sha1 /opt/cloudera/parcel-repo/CDH-5.8.5-1.cdh5.8.5.p0.5-el7.parcel.sha
cp /home/manifest.json /opt/cloudera/parcel-repo/
cdh1 starts the server side:
/opt/cm-5.8.5/etc/init.d/cloudera-scm-server start
cdh[1-3] Start the Agent:
/opt/cm-5.8.5/etc/init.d/cloudera-scm-agent start
It takes 2-5 minutes to start up
Address: http://localhost:7180
User: admin
Password: admin
Summary of common problems
1. Check the correctness of the host and the solution to the warning
Execute the following commands in the three servers
sysctl -w vm.swappiness=10
vim /etc/sysctl.conf
vm.swappiness=10
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
vim /etc/rc.local
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
2. The database setting hue database cannot be connected
Execute the command on the cdh2 and cdh3 nodes
scp -r /usr/lib64/mysql/libmysqlclient.so.18 root@cdh1:/usr/lib64
或者参考
https://blog.csdn.net/ckg8933/article/details/83379279
3. Hive lacks mysql driver
cdh1 execution
cp /home/mysql-connector-java-5.1.42-bin.jar /opt/cloudera/parcels/CDH-5.8.5-1.cdh5.8.5.p0.5/lib/hive/lib/
4. Oozie lacks mysql driver
cdh1 execution
cp /home/mysql-connector-java-5.1.42-bin.jar /opt/cloudera/parcels/CDH-5.8.5-1.cdh5.8.5.p0.5/lib/oozie/lib/
cp /home/mysql-connector-java-5.1.42-bin.jar /var/lib/oozie/mysql-connector-jave.jar
chmod 777 /var/lib/oozie/mysql-connector-jave.jar
5. The hue database test connection failed
https://blog.csdn.net/ckg8933/article/details/83379279
6. The database test connection fails
https://blog.csdn.net/levy_cui/article/details/51142843