Centos7 online installation CDH6

The following steps are valid and correct! ! ! ! !

1. Basic environment preparation

1) Software list 3
Centos7 (64-bit) servers, the address is as follows:
jdk1.8
Mysql5.7 (version 5.7 is required!!)
mysql-connector-java-5.1.42-bin.jar
cloudera-manager-agent- 6.3.1-1466458.el7.x86_64.rpm (https://archive.cloudera.com/cm6/6.3.1/redhat7/yum/RPMS/x86_64/)
cloudera-manager-daemons-6.2.0-968826.el7 .x86_64.rpm(https://archive.cloudera.com/cm6/6.3.1/redhat7/yum/RPMS/x86_64/)
cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm(https: //archive.cloudera.com/cm6/6.3.1/redhat7/yum/RPMS/x86_64/)
oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm (https://archive.cloudera.com /cm6/6.3.1/redhat7/yum/RPMS/x86_64/)
CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel (https://archive.cloudera.com/cdh6/6.3. 2/parcels/)
CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1（https://archive.cloudera.com/cdh6/6.3.2/parcels/）
manifest.json（https://archive.cloudera.com/cdh6/6.3.2/parcels/）

2) Server address
192.168.56.1 (this address maps your own server address)
192.168.56.2 (this address maps your own server address)
192.168.56.3 (this address maps your own server address)

3) Modify hostname

vim /etc/hostname

After modifying the host name cdh*, do the following

service network restart

4) Configure hosts

vim /etc/hosts

192.168.56.1 cdh1
192.168.56.2 cdh2
192.168.56.3 cdh3

5) Secret-free access

ssh-keygen -t rsa
//三台服务器全都执行以下命令
ssh-copy-id  root@cdh1
ssh-copy-id  root@cdh2
ssh-copy-id  root@cdh1

6) Turn off the firewall

systemctl stop firewalld
systemctl disable firewalld

7) Close SELINUX

setenforce 0
vim /etc/sysconfig/selinux
SELINUX=disabled

8) Configure NTP

yum -y install ntp
systemctl start ntpd 
systemctl enable ntpd 
在cdh1节点
vim /etc/ntp.conf
server 0.cn.pool.ntp.org
server 1.cn.pool.ntp.org
server 2.cn.pool.ntp.org
server 3.cn.pool.ntp.org
restrict 0.cn.pool.ntp.org nomodify notrap noquery
restrict 1.cn.pool.ntp.org nomodify notrap noquery
restrict 2.cn.pool.ntp.org nomodify notrap noquery
restrict 3.cn.pool.ntp.org nomodify notrap noquery
server 127.127.1.0
fudge 127.127.1.0 stratum 10

systemctl start ntpd.service
systemctl enable ntpd.service
ntpstat
在cdh[2-3]节点
vim /etc/ntp.conf
server cdh1 prefer

systemctl start ntpd.service
systemctl enable ntpd.service
ntpdate -u cdh1
ntpstat

9) Install JDK

mkdir /usr/local/java
cp /home/ jdk-8u131-linux-x64.gz /usr/local/java/
cd /usr/local/java/
tar xvf jdk-8u131-linux-x64.gz
rm jdk-8u131-linux-x64.gz
scp -r /usr/local/java/ root@cdh2:/usr/local/
scp -r /usr/local/java/ root@cdh3:/usr/local/

Set environment variables

vi /etc/profile
export JAVA_HOME=/usr/local/jdk
export JRE_HOME=$JAVA_HOME/jre
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$PATH

java -version

10) Install MySQL

Delete your own database

rpm -qa | grep mariadb
rpm -e –nodeps 包名
rm -rf /etc/my.cnf

Download dependencies and create users and groups

yum -y install perl perl-devel autoconf
groupadd mysql
useradd -g mysql mysql
id mysql

Please refer to the following link for installation:

https://blog.csdn.net/baidu_32872293/article/details/80557668
https://www.cnblogs.com/downmoon/p/11896158.html

Set the database to allow remote login

mysql -u root -p
use mysql;
update user set host=’%’ where user=’root’;

If the error is reported, it may have been set up, use the following statement to view

select host from user where user = ‘root’;
flush privileges;

Change the database character driver

set character_set_client=utf8;
set character_set_connection=utf8;
set character_set_results=utf8;
set character_set_server=utf8;
set character_set_system=utf8;

View modification results

show variables like ‘character_set_%’;

Close the database security mode, so that the database can be added, deleted, and modified

SET SQL_SAFE_UPDATES=0;

Create a database and make the local root user have access

mysql -u root -p
create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database am DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database oozie DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
grant all privileges on *.* to ‘root’@’cdh1’ identified by ‘000000’ with grant option;
flush privileges;

Install CDH Server (master node)

Prepare in advance:

将mysql-connector-java-5.1.42-bin.jar改名为mysql-connector-java.jar放入/usr/share/java目录下

Database modification /etc/my.cnf

[mysqld]
validate_password=off
wait_timeout=31536000
interactive_timeout=31536000

Install cloudera rpm

yum install oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm

yum install cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm

Parcel initialization
new parcel directory

mkdir -p /opt/cloudera/parcel-repo

Copy the installation package file to the default installation path

cp /home/CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel /opt/cloudera/parcel-repo/
cp /home/CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1 /opt/cloudera/parcel-repo/CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha
cp /home/manifest.json /opt/cloudera/parcel-repo/

Execute the create database command

/opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm -hlocalhost  -uroot -p'000000' --scm-host localhost scm scm scm

Restart cloudera-scm-server

service cloudera-scm-server restart

Address: http://localhost:7180
User: admin
Password: admin

Install CDH Agent (each node)

Prepare in advance:

将mysql-connector-java-5.1.42-bin.jar改名为mysql-connector-java.jar放入/usr/share/java目录下

Install cloudera rpm

yum install oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm

yum install  cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm

Modify agent configuration

vim /etc/cloudera-scm-agent/config.ini
server_host=cdh1

Start agent

systemctl start cloudera-scm-agent

Summary of common problems

1. Check the correctness of the host and the solution to the warning

Execute the following commands in the three servers

sysctl -w vm.swappiness=10
vim /etc/sysctl.conf
vm.swappiness=10

echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
vim /etc/rc.local
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled

2. The database setting hue database cannot be connected

Execute the command on the cdh2 and cdh3 nodes

scp -r /usr/lib64/mysql/libmysqlclient.so.18 root@cdh1:/usr/lib64

或者参考
https://blog.csdn.net/ckg8933/article/details/83379279

3. Hive lacks mysql driver

cdh1 execution

cp /home/mysql-connector-java-5.1.42-bin.jar /opt/cloudera/parcels/CDH-5.8.5-1.cdh5.8.5.p0.5/lib/hive/lib/

4. Oozie lacks mysql driver

cdh1 execution

cp /home/mysql-connector-java-5.1.42-bin.jar /opt/cloudera/parcels/CDH-5.8.5-1.cdh5.8.5.p0.5/lib/oozie/lib/
cp /home/mysql-connector-java-5.1.42-bin.jar /var/lib/oozie/mysql-connector-jave.jar
chmod 777 /var/lib/oozie/mysql-connector-jave.jar

5. The hue database test connection failed

https://blog.csdn.net/ckg8933/article/details/83379279

6. The database test connection fails

https://blog.csdn.net/levy_cui/article/details/51142843

7. Cannot find the local parcle library

chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo/*
chmod 777 /opt/cloudera/parcel-repo/*

重启服务

8. HDFS folder permission problem

chmod 777 -R nn
chmod 777 -R snn
chmod o-w -R dn
chmod o-w -R var

9. The lack of libmysqlclient.so.18()

下载安装此组件
https://download.csdn.net/download/lufengbo123/12195518

10. Various abnormalities occur in the use of hbase

删除zookeeper下hbase目录，然后重启hbase即可

If the Master is initializing error still occurs, after deleting the hbase directory in zookeeper, stop the hbase cluster and use hdfs dfs -rmr /hbase to
delete the hdfs file, create a new hbase root directory on the front end of hbase and restart.

11. Before starting

看看各个组件内部是否需要创建文件夹和数据库

12. Use hbase to report errors in hue

Hbase中取消以下两个选项

hue中添加以下选项
[hbase]
thrift_transport=buffered