Progress a little bit every day - CDH and CM construction

Get into the habit of writing together! This is the 10th day of my participation in the "Nuggets Daily New Plan·April Update Challenge", click to view the details of the event

1. Preface

The previous understanding of CDH and CM is only a literal understanding. The installation of CDH is a temporary initiative, so the installation location is 5 KVM virtual machines that are restarted on the Linux server.

Record it.

2. Prepare

Installation version and download address:

JDK环境:
    版本:jdk-8u171-linux-x64.tar.gz
    地址:官网..
    
mysql:
    版本:mysql-5.6.40-linux-glibc2.12-x86_64.tar.gz
    地址:官网...
    
    jdbc连接包mysql-connector-java.jar: 
        http://dev.mysql.com/downloads/connector/j/
    

CM:
    版本:5.15.1
    地址:http://archive.cloudera.com/cm5/cm/5/cloudera-manager-centos7-cm5.15.1_x86_64.tar.gz
        
CDH:
    版本:5.15.1
    地址:http://archive.cloudera.com/cdh5/parcels/5.15.1/
        三个文件:
            CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel
            CDH-5.15.1-1.cdh5.15.1.p0.4-el7.parcel.sha1
            manifest.json
    
操作系统:
    版本:Centos7.5
    地址:http://mirrors.aliyun.com/centos/7.5.1804/isos/x86_64/CentOS-7-x86_64-Everything-1804.iso
    
复制代码

system environment

具体开设KVM的步骤在此就不补充了，详情见{Linux系统KVM虚拟化}
复制代码

machine allocation

IP address	CPU name	illustrate	configure
192.168.0.111	pazi1	master node	16G+50G
192.168.0.112	pazi2	slave node	4G + 50G
192.168.0.113	pazi3	slave node	4G + 50G
192.168.0.114	p44	slave node	4G + 50G
192.168.0.115	pazi5	slave node	4G + 50G

---

3. Environment setup before installation (all nodes)

• yum source configuration

  Since all virtual machines can be connected to the Internet, Alibaba Cloud's yum online yum source is used here.

• Network configuration: All cluster machine IP configuration and host name host configuration.

   配置文件：
        ip：  /etc/sysconfig/network-config/xxx
        host：/ect/hosts
  复制代码

• SSH password-free login:

    使用ssh免密配置时，需要系统里面已经安装了openssh依赖包；
        
    以下命令逐行执行：
            ssh-keygen -t rsa (一路回车即可)
            ssh-copy-id {hostname} (拷贝{hostname}的公钥到本地)
复制代码

• Server firewall turned off

    Centos6和7使用默认的防火墙不一样，6*使用的是iptables，7*使用的是firewalld。
    
    Centos6.* ：
            临时关闭：service iptables stop
            开机禁用：chkconfig iptables off
    Centos7.* :
            临时关闭：systemctl stop firewalld
            开机禁用：systemctl disable firewalld
复制代码

• SELinux off

    临时关闭：
        setenforce 0
    永久关闭:
        修改配置文件/etc/selinux/config（重启生效）：
            将 SELINUX=enforcing 改为 SELINUX=disabled
        查看 SELINUX 状态：
            1)  /usr/sbin/sestatus –v
                SELinux status: enabled（enabled：开启；disabled：关闭）
            2) 使用命令：getenforce
复制代码

• Install JDK

    
    此处省略....
复制代码

• Set up NTP

    服务器联网环境或者已经配置好本地yum源:
    
    安装：
        yum install ntp
    配置开机自启：
        chkconfig ntpd on
    检查是否成功：
        chkconfig --list ntpd （2-5 为 on 状态则成功）
    设置同步：
        // 此处设置的同步是同步与210.72.145.44-国家授时中心服务器
        // 一般的集群环境中同步的为主节点的时间
        ntpdate -u ntp.sjtu.edu.cn
    查看服务状态：
        service ntpd status
        
    
    配置同步服务器主节点时间：
        详情见{Linux常用操作}-->{NTP时间同步}
复制代码

• Install the corresponding dependencies

    yum -y install zlib zlib-devel bzip2 bzip2-devel ncurses ncurses-devel readline readline-devel openssl* xz lzma xz-devel gdbm gdbm-devel tk tk-devel glib glib-devel glibc glibc-devel glibc-static gcc gcc-c++ numactl numactl-devel openssh* perl perl-devel gd gd-devel libaio* autoconf numactl libffi-devel libuuid-devel vim wget screen net-tool
复制代码

---

4. CM installation

Unzip and install Cloudera Manager Server&Agent

# 此操作是所有节点操作

创建安装CM的文件夹,将下载的tar包解压.

[root@pazi1 ~]# mkdir /opt/cloudera-manager
[root@pazi1 ~]# tar -zxvf cloudera-manager*.tar.gz -C /opt/cloudera-manager  
复制代码

Create cloudera-scm user

# 此操作是所有节点操作

[root@pazi1 ~]# useradd --system --home=/opt/cloudera-manager/cm-5.15.1/run/cloudera-scm-server --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm  
复制代码

Configure the CM Agent to point to the master node

# 此操作是所有子节点操作

[root@pazi1 cloudera-manager]# vim /opt/cloudera-manager/cm-5.15.1/etc/cloudera-scm-agent/config.ini

将server_host改为主节点的主机名即可.
复制代码

Configure the database of the CM Server

# 初始脚本配置数据库scm_prepare_database.sh
* 在初始化之前,需要将链接mysql的驱动包复制到对应文件夹.
* 驱动包的名字一定要叫这个,否则后续初始化的时候会报错.
[root@pazi1 cloudera-manager]# cp mysql-connector-java-5.1.31-bin.jar /usr/share/java/mysql-connector-java.jar


# 此操作是在主节点上

* 执行下列脚本
[root@pazi1 cloudera-manager]# /opt/cloudera-manager/cm-5.15.1/share/cmf/schema/scm_prepare_database.sh mysql -hpazi1 -uroot -proot --scm-host pazi1 scmdbn scmdbu root
说明：
    这个脚本就是用来创建和配置CMS需要的数据库的脚本。
各参数是指：
    mysql：数据库用的是mysql，如果安装过程中用的oracle，那么该参数就应该改为oracle。
    -hpazi1：数据库建立在cdh1主机上面,也就是主节点上面。
    -uroot：root身份运行mysql.
    -proot: root用户的密码   
    ## 下面这个配置之前遇到过坑,有时候需要填写主机的IP才可以.至于为啥,未知....
    --scm-host pazi1：CMS的主机，一般是和mysql安装的主机是在同一个主机上.
    最后三个参数是：数据库名，数据库用户名，数据库密码。
复制代码

Create Parcel directory

    主节点:
        [root@pazi1 ~]#  mkdir -p /opt/cloudera/parcel-repo
        [root@pazi1 ~]#  chown cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo
        
        // 此处的`CDH-*.parcel.sha`文件是下载的`CDH-*.parcel.sha1`文件重命名得来的.
        将下载好的文件（CDH-*.parcel、CDH-*.parcel.sha、manifest.json）拷贝到该目录下。
    
    子节点:
        [root@pazi2 ~]#  mkdir -p /opt/cloudera/parcels
        [root@pazi2 ~]#  chown cloudera-scm:cloudera-scm /opt/cloudera/parcels
复制代码

Start the CM service

主节点:
    [root@pazi1 ~]# /opt/cloudera-manager/cm-5.15.1/etc/init.d/cloudera-scm-server start
主节点以及子节点:
    [root@pazi2 ~]# /opt/cloudera-manager/cm-5.15.1/etc/init.d/cloudera-scm-agent start

其余操作:
    关闭:
        将start修改为:stop
    查看状态:
        将start修改为:status

启动完毕之后:
    访问：http://pazi1:7180，若可以访问（用户名、密码：admin） ，则安装成功。

注意:
    Manager启动成功需要等待一段时间，过程中会在数据库中创建对应的表需要耗费一些时间。
复制代码

5. CDH installation

1. Visit http://192.168.0.111:7180 , enter the username and password (admin/admin), and click "Login"

2, will first point out the different functions of different versions

3. Select the host where CDH is to be installed.

Configure the host: Since we have installed and started the agent on each node, and each node points to pazi1 as the server node in the configuration file, here we can see three hosts in the "currently managed hosts", all checked Select and continue.

Note: If cloudera-scm-agent is not set to start on boot, other servers may not be detected if the above restarts.

4. Select the version of CDH to install, here we have to select the version we downloaded and uploaded to Linux

The stage of downloading and installing the configuration will be slow, and you need to wait patiently

5. Check the host

After checking, two warnings will appear, the solution:

The first warning is executed directly on all nodes

echo 10 > /proc/sys/vm/swappiness

The second warning follow the instructions above, execute these two commands on all nodes

echo never > /sys/kernel/mm/transparent_hugepage/defrag >> /etc/rc.local

echo never > /sys/kernel/mm/transparent_hugepage/enabled >> /etc/rc.local

6. Cluster settings

Select the components to install

Assign component node relationships

Configure database links for related components

Configure some configuration information of related components and various storage directories (default)

It can also be recorded, and it may be used later.

It is time-consuming and error-prone to start installing each component. Please refer to the error log after an error occurs.