CDH6.2 offline installation (reprint edition)

1 Overview

　　CDH, the full name of Cloudera's Distribution, including Apache Hadoop. Hadoop is one of the many branches of the corresponding maintained by Cloudera, built on a stable version of Apache Hadoop, provides the core of Hadoop (scalable storage, distributed computing), the most important thing is to provide web-based user interface.

　　CDH advantage of: Versioning clear, update speed, support for Kerberos security authentication, support a variety of installation methods (such as Yum, rpm, etc.).

　　CDH into Cloudera Manager management platform and CDH parcel (parcel various components comprising the installation package). Used here CDH6.2.0.

　　Cloudera Manger download address https://archive.cloudera.com/cm6/6.2.0/redhat7/yum/RPMS/x86_64/ , as shown:

　　

 　　CDH6.2.0 installation package address: https://archive.cloudera.com/cdh6/6.2.0/parcels/

 　　Because the operating system is CentOS7, you need to download the following files:

　　

　　The above-mentioned document collating information Baidu cloud download address is:

　　Links: https://pan.baidu.com/s/1Dm5Elf9uQqn14BUbgU3AFQ extraction code: mws3 

Installation procedure

Description: The following operations are performed in the root user

2.1 Environment Preparation

1. Prepare a virtual machine (according to their own virtual machine resource allocation system resources)

 

2. Static IP settings (each node)

vim /etc/sysconfig/network-scripts/ifcfg-ens192

service network restart to restart the network into force

yum install -y net-tools ifconfig disposed See

3. Edit / etc / hosts file (for each node)

vim /etc/hosts

4. 关闭防火墙、禁止防火墙开机自启（每个节点）

• systemctl stop firewalld 关闭防火墙
• systemctl disable firewalld 禁止防火墙开机自启
• vim /etc/selinux/config —> SELINUX=disabled (修改)

5. ssh无密码登录

• manager节点执行ssh-keygen -t rsa 一路回车到结束，在/root/.ssh/下面会生成一个公钥文件id_rsa.pub
• cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys 将公钥追加到authorized_keys
• chmod 600 ~/.ssh/authorized_keys 修改权限
• 将~/.ssh从当前节点分发到其他各个节点。如：scp -r ~/.ssh/ root@node1:~/.ssh/
• ssh 各个节点互相登陆

6. 配置NTP服务（所有节点）

• 修改时区（改为中国标准时区）ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
• 安装ntp yum -y install ntp
• ntp主机配置 vim /etc/ntp.conf
• manager节点

其余节点

• 重新启动 ntp 服务：service ntpd restart
• 设置开机自启：systemctl enable ntpd.service
• ntpdc -c loopinfo #查看与时间同步服务器的时间偏差
• ntpq -p #查看当前同步的时间服务器
• ntpstat #查看状态
• 配置成功状态（服务开启后前面出现*说明成功）：

7. 修改Linux swappiness参数(所有节点）

为了避免服务器使用swap功能而影响服务器性能，一般都会把vm.swappiness修改为0（cloudera建议10以下）

• 上述方法rhel6有效，rhel7.2中:tuned服务会动态调整系统参数
• 查找tuned中配置，直接修改配置
• cd /usr/lib/tuned/
• grep “vm.swappiness” * -R 查询出后依次修改

修改完确认:

参考：sysctl修改内核参数 重启不生效 https://blog.csdn.net/ygtlovezf/article/details/79014299

8. 禁用透明页(所有节点）

echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
永久生效 在/etc/rc.local 添加上面命令

 

给与可执行权限：chmod +x /etc/rc.d/rc.local

9. JDK安装（所有节点）

• rpm -qa | grep java # 查询已安装的java
• yum remove java* # 卸载
• rpm -ivh oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm
• vi /etc/profile 末尾添加

• source /etc/profile
• java -version验证

10. 创建/usr/share/java目录，将mysql-jdbc包放过去（所有节点）

• mkdir -p /usr/share/java
• mv /opt/mysql-j/mysql-connector-java-5.1.34.jar /usr/share/java/
• mysql-connector-java-5.1.34.jar 一定要命名为mysql-connector-java.jar

11. 为保证防火墙、虚拟机参数修改后生效，各节点机器需要重启 reboot

12. Mysql安装

• 卸载mariadb：rpm -qa|grep mariadb
• rpm -e --nodeps mariadb-libs-5.5.60-1.el7_5.x86_64
• cd /opt/mysql/
• tar -xvf ./mysql-5.7.19-1.el7.x86_64.rpm-bundle.tar
• rpm -ivh mysql-community-common-5.7.19-1.el7.x86_64.rpm
• rpm -ivh mysql-community-libs-5.7.19-1.el7.x86_64.rpm
• rpm -ivh mysql-community-client-5.7.19-1.el7.x86_64.rpm
• rpm -ivh mysql-community-server-5.7.19-1.el7.x86_64.rpm
• rpm -ivh mysql-community-libs-compat-5.7.19-1.el7.x86_64.rpm
• MYSQL配置:
• mysqld --initialize --user=mysql # 初始化mysql使mysql目录的拥有者为mysql用户
• cat /var/log/mysqld.log # 最后一行将会有随机生成的密码
• systemctl start mysqld.service # 设置mysql服务自启
• mysql -uroot –p 如果不能登陆
• 设置免密登录并重启mysql服务
• vi /etc/my.cnf
• systemctl restart mysqld
• #登录

　　　　$>mysql -u root
　　　　mysql>use mysql;
　　　　mysql>update user set authentication_string = password(‘123456’), password_expired = ‘N’, password_last_changed = now() where user = ‘root’;
　　　　mysql>exit;

• 创建库(后续安装服务等使用)

13. 安装Httpd服务（manager）

• yum install httpd
• service httpd start
• systemctl enable httpd.service 设置httpd服务开机自启

14. 防止出现莫名错误，事先安装以下插件（所有节点）

• yum install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server --skip-broken --nogpgcheck

15. 配置Cloudera Manager包yum源（manager节点）

• mkdir -p /var/www/html/cloudera-repos/
• 将下载的cm包文件移到此目录下:
• mv cm6 /var/www/html/cloudera-repos/
• cd /var/www/html/cloudera-repos/cm6/
• 创建repodata： createrepo .

• vim /etc/yum.repos.d/cloudera-manager.repo

　　注意路径：http://manager/cloudera-repos/cm6/

• yum clean all
• yum makecache

16.导入GPG key（如果没有这步操作，很可能cloudera服务安装失败）manager节点

• rpm --import https://archive.cloudera.com/cm6/6.2.0/redhat7/yum/RPM-GPG-KEY-cloudera

17. 安装 Cloudera Manager（manager节点）

• sudo yum install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server
• 安装完CM后/opt/ 下会出现cloudera目录
• mv /opt/parcels/* /opt/cloudera/parcel-repo # 将parcel包移动到指定位置
• 在/opt/cloudera/parcel-repo执行以下命令：
• sha1sum CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel | awk ‘{ print $1 }’ > CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel.sha

• 执行初始化脚本:
• /opt/cloudera/cm/schema/scm_prepare_database.sh mysql cmserver cmserveruser password
• 打开server服务:
• service cloudera-scm-server start
• 静候几分钟，打开http://manager:7180

总结

【参考资料】

https://blog.csdn.net/wolf_333/article/details/89071203

http://www.cnblogs.com/mylovelulu/p/10384732.html

https://blog.csdn.net/qq_40127822/article/details/84441869

https://www.cnblogs.com/raphael5200/p/5293960.html