CDH 5.16.1 & offline deployment by deploying Hadoop CDH service

Server 2019-07-31 08:10:44 views: null

A. Environmental

1.1 System node information

Hostname IP CPU(cores) Memory(GB) THE Service Remark
master 172.30.200.75 2 4 7.5 jdk, cloudera-scm-server, mysql After the actual deployment, master nodes need to be deployedcloudera-scm-agent服务，或采用4个slave节点
slave01 172.30.200.76 2 4 CentOS 7.5 jdk, cloudera-scm-agent  
slave02 172.30.200.77 2 4 CentOS 7.5 jdk, cloudera-scm-agent  
slave03 172.30.200.78 2 4 CentOS 7.5 jdk, cloudera-scm-agent  

Software Version 1.2

Related software placed in/usr/local/src/目录。

Soft Version Download Remark
CM(Cloudera Manager) cloudera-manager-centos7-cm5.16.1_x86_64.tar.gz bin bags, depending on the version downloaded  
CDH parcel CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel Depending on the version Download Software installation package
CDH parcel.sha CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.sha1 Depending on the version Download Packages hash code
CDH manifest.json manifest.json Depending on the version Download Imprint file
JDK jdk-8u181-linux-x64.tar.gz bin bags, depending on the version downloaded  
MySQL mysql-5.7.24-el7-x86_64.tar.gz bin bags, depending on the version downloaded Cloudera Manager configuration file is stored
MySQL-connector-Java mysql-connector-java-8.0.13.jar jar package, depending on the version downloaded JDBC

II. Preparations

2.1 part preconfigured

All nodes do the following:

  • Permanently turn off the firewall (firewalld或iptables)；
  • Permanently closedselinux；
  • Set upntp。

2.2 Setting hosts

  • All nodes set up / etc / hosts

    cat << EOF >> /etc/hosts

#  nodes
172.30.200.75 master
172.30.200.76 slave01
172.30.200.77 slave02
172.30.200.78 slave03
EOF

Free access is provided secret 2.3

  • All nodes generated secret key

    ssh-keygen -t rsa

  • In the master section generating pointauthorized_keys文件

    cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys

  • Master node performs in the cluster each node's public keyid_rsa.pub放入master节点的authorized_keys文件中

    for i in {1..3}; do ssh root@slave0$i cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys; done
  • In the master node performs the master nodeauthorized_keys文件放置到集群每个节点的/root/.ssh/目录，依然命名位authorized_keys
    • The net effect is that all the nodes in the cluster density twenty-two Free access to each other;
    • Note : the first time landing公钥检查，可通过在/etc/ssh/ssh_config文件中设置StrictHostKeyChecking no绕过，或使用-o参数跳过。
    for i in {1..3}; do scp /root/.ssh/authorized_keys root@slave0$i:/root/.ssh/authorized_keys; done

Install JDK 2.4

All nodes need to install the JDK.

  • Install the JDK

    cd / usr / local / src 
tar -zxvf the JDK-8u181-Linux-x64.tar.gz 
mkdir -p / usr / the Java 
mv jdk1.8.0_181 / / usr / the Java / 

# Set soft links, easy to upgrade to replace 
ln -s /usr/java/jdk1.8.0_181/ / usr / java / current

  • Set the variable

    cat << EOF >> /etc/profile

# JDK
export JAVA_HOME=/usr/java/current
export JRE_HOME=/usr/java/current/jre
export PATH=$PATH:/usr/java/current/bin
export CLASSPATH=./:/usr/java/current/lib:/usr/java/current/jre/lib
EOF

  • verification

    # Load variable 
Source / etc / Profile 

# verify 
java -version

2.5 Installing MySQL

Only master node need to install MySQL.

  • Unloading system comes with a relational database

    -qa RPM | grep MySQL 
RPM -qa | grep MariaDB 

# - nodeps: Do not check dependence 
rpm -e --nodeps mariadb-libs-5.5.56-2.el7.x86_64

  • MySQL deployment

    # Bin bag deployment using, directly after decompression 
CD / usr / local / the src 
the tar--zxvf MySQL 5.7.24-EL7-x86_64.tar.gz -C / usr / local / 
Music Videos /usr/local/mysql-5.7. 24-el7-x86_64 / / usr / local / mysql

  • Accounts and permissions

    Add Account # 
groupadd MySQL 
useradd -g MySQL -s / sbin / nologin MySQL 

# Create a "data" directory 
mkdir / the Data 

# empowerment 
chown -R MySQL: MySQL / usr / local / MySQL / 
chown -R MySQL: MySQL / the Data /
  • MySQL initialization
    • Obtainroot@localhost账号的初始密码Hqe6x<Re4jhK
    cd /usr/local/mysql/
bin/mysqld --initialize --user=mysql --basedir=/usr/local/mysql --datadir=/data

  • Set the variable

    << EOF >> CAT / etc / Profile 

# MySQL 
Export the PATH = $ the PATH: / usr / local / MySQL / bin 
EOF 

# load variable 
Source / etc / Profile 

# soft link 
ln -s / usr / local / mysql / lib / MySQL / usr / lib / MySQL 
LN -s / usr / local / MySQL / the include / MySQL / usr / the include / MySQL

  • Set boot

    # Copy startup script to system services 
cp /usr/local/mysql/support-files/mysql.server /etc/rc.d/init.d/mysqld 
chown MySQL: MySQL /etc/rc.d/init.d/ mysqld 

# modify the default "basedir" and "datadir" 
vim /etc/rc.d/init.d/mysqld 
basedir = / usr / local / MySQL 
datadir = / the Data 

# add startup script 
chkconfig --add mysqld 
chkconfig - level 35 mysqld on

  • File path: log && pid && socket

    # Log path 
mkdir -p / var / log / mysqld 
Touch /var/log/mysqld/mysqld.log 
chown -R & lt MySQL: MySQL / var / log / mysqld / 

# PID path 
mkdir -p / var / RUN / mysqld 
chown - MySQL R & lt: MySQL / var / RUN / mysqld / 

# Socker path 
mkdir -p / var / lib / mysqld 
chown -R & lt MySQL: MySQL / var / lib / mysqld / 
LN -s /var/lib/mysqld/mysql.sock / tmp / mysql.sock

  • Set my.cnf file

    # 注意"log-error"，"pid-file"与"socket"的路径
mkdir -p /usr/local/mysql/etc
cat << EOF >> /usr/local/mysql/etc/my.cnf
[mysqld]
character-set-server=utf8
max_connections = 3000
log-error=/var/log/mysqld/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
socket=/var/lib/mysqld/mysql.sock
sql_mode=NO_ENGINE_SUBSTITUTION,STRICT_TRANS_TABLES

[mysql]
default-character-set=utf8
EOF

# 软链接
ln -s /usr/local/mysql/etc/my.cnf /etc/my.cnf

# 赋权
chown -R mysql:mysql /usr/local/mysql/etc/

  • Start MySQL service

    # Start Service 
Service Start mysqld 

# verification 
service mysqld status

  • Set MySQL account password and login privileges

    # Initialize password using 
MySQL-uroot--p 

# change your password, be careful not to use the "$" and other special symbols 
the SET password = password ( 'CDH12 # hadoop'); 
flush privileges; 

# remote access permissions 
grant all privileges on * * to. 'root' @ '%' IDENTIFIED by 'CDH12 # hadoop' with the Option Grant; 
flush privileges; 

# View account 
select user, host, authentication_string from mysql.user ;

Three. Cloudera Manager Installation

3.1 deployment CM Server & Agent

  • In all nodes to create a directory CM/opt/cloudera-manager

    mkdir /opt/cloudera-manager

  • In the master node distributed to the remaining nodescloudera-manager-centos7-cm5.16.1_x86_64.tar.gz

    for i in {1..3}; do scp /usr/local/src/cloudera-manager-centos7-cm5.16.1_x86_64.tar.gz  root@slave0$i:/usr/local/src/ ; done

  • In all nodes decompressioncloudera-manager-centos7-cm5.16.1_x86_64.tar.gz到/opt/cloudera-manager/目录

    tar -zxvf /usr/local/src/cloudera-manager-centos7-cm5.16.1_x86_64.tar.gz -C /opt/cloudera-manager/

  • In all slave nodes modifications/opt/cloudera-manager/cm-5.16.1/etc/cloudera-scm-agent/config.ini的server_host参数为master节点ip或主机名

    sed -i "s/server_host=localhost/server_host=master/g" /opt/cloudera-manager/cm-5.16.1/etc/cloudera-scm-agent/config.ini

3.2 && account permissions

  • In all the nodes createdcloudera-scm账号，这是CM相关服务使用的默认账号

    # Prohibit the use of "cloudera-scm" account login 
useradd --system --home = / opt / cloudera -manager / cm-5.16.1 / run / cloudera-scm-server / --no-create-home --shell = / bin / false --comment "Cloudera SCM User" cloudera-scm

  • In all nodes is/opt/cloudera-manager目录赋权

    chown -R cloudera-scm:cloudera-scm /opt/cloudera-manager

3.3 Setting the boot

  • In the master node set up system services

    # Set to use "cloudera-scm-server" to start the service for the system 
cp /opt/cloudera-manager/cm-5.16.1/etc/init.d/cloudera-scm-server /etc/rc.d/init.d/ 
SCM-Cloudera chown: Cloudera-SCM /etc/rc.d/init.d/cloudera-scm-server 

# Review "CMF_DEFAULTS = $ {CMF_DEFAULTS: - / etc / default}" path 
vim /etc/rc.d/ the init.d / Cloudera-SCM-Server 
CMF_DEFAULTS = / opt / Cloudera-Manager / cm & lt-5.16.1 / etc / default 


# added service system starts 
the chkconfig --add Cloudera-SCM-Server 
the chkconfig --level 35 Cloudera-scm- ON Server 
checkconfig --list

  • In all slave nodes set up system services

    # Set to use "cloudera-scm-agent" for the system to start the service 
cp /opt/cloudera-manager/cm-5.16.1/etc/init.d/cloudera-scm-agent /etc/rc.d/init.d/ 
SCM-Cloudera chown: Cloudera-SCM /etc/rc.d/init.d/cloudera-scm-agent 

# Review "CMF_DEFAULTS = $ {CMF_DEFAULTS: - / etc / default}" path "- / etc / default" 
Vim /etc/rc.d/init.d/cloudera-scm-agent 
CMF_DEFAULTS = $ {CMF_DEFAULTS: /opt/cloudera-manager/cm-5.16.1/etc/default} 

# Add the system to start the service 
chkconfig --add cloudera- Agent-SCM 
the chkconfig --level Cloudera-35 SCM-Agent ON 
checkconfig --list

3.4 initialize the database

  • All nodes disposed MySQL driver (the JDBC);

  • Note : JDBC deployed in any node, then the subsequent "Configuration of CDH Installation" phaseReports Manager被分配在任意节点都可以

    cp /usr/local/src/mysql-connector-java-8.0.13.jar /opt/cloudera-manager/cm-5.16.1/share/cmf/lib/
chown cloudera-scm:cloudera-scm /opt/cloudera-manager/cm-5.16.1/share/cmf/lib/mysql-connector-java-8.0.13.jar

  • In the master node to restart the MySQL service

    service mysqld restart
  • In any node initialization CM
  • Note : Cloudera related service needsdatabase如下：
    • CM is given in the table related services in the default configuration filedatabase与user，但不是必须使用；
    • database在数据库中可直接创建，但CM初始化时如果没有database，则自动创建。
    Service Database User
    Cloudera Manager Server scm scm
    Activity Monitor amon amon
    Reports Manager rman rman
    Hue hue hue
    Hive Metastore Server metastore metastore
    Sentry Server sentry sentry
    Cloudera Navigator Audit Server are not are not
    Cloudera Navigator Metadata Server navms navms
    Oozie oozie oozie 
    # 格式：scm_prepare_database.sh [options] (postgresql|mysql|) database username [password]
# scm_prepare_database.sh：创建与配置CMS需要的数据库脚本，默认在"/opt/cloudera-manager/cm-5.16.1/share/cmf/schema/"目录；
# postgresql|mysql|oracle：必选项，数据库类型；
# database：必选项，针对postgresql|mysql，创建SCM数据库;针对oracle，填写sid；
# username：必选项，SCM数据库的账号；
# password：选填项，SCM数据库的账号密码，如果不指定，会提示输入；
# options：
# -h：数据库主机ip或hostname，默认是"localhost"；
# -u：数据库账号，需要具备增删改查的权限，默认是"root"；
# -p：账号密码，默认无密码；
# --scm-host：SCM server主机名，默认是"localhost"
/opt/cloudera-manager/cm-5.16.1/share/cmf/schema/scm_prepare_database.sh mysql -h master -uroot -pcdh12#hadoop --scm-host % scm scm scm_pass
/opt/cloudera-manager/cm-5.16.1/share/cmf/schema/scm_prepare_database.sh mysql -h master -uroot -pcdh12#hadoop --scm-host % amon amon amon_pass
/opt/cloudera-manager/cm-5.16.1/share/cmf/schema/scm_prepare_database.sh mysql -h master -uroot -pcdh12#hadoop --scm-host % rman rman rman_pass
/opt/cloudera-manager/cm-5.16.1/share/cmf/schema/scm_prepare_database.sh mysql -h master -uroot -pcdh12#hadoop --scm-host % hue hue hue_pass
/opt/cloudera-manager/cm-5.16.1/share/cmf/schema/scm_prepare_database.sh mysql -h master -uroot -pcdh12#hadoop --scm-host % metastore metastore metastore_pass
/opt/cloudera-manager/cm-5.16.1/share/cmf/schema/scm_prepare_database.sh mysql -h master -uroot -pcdh12#hadoop --scm-host % sentry sentry sentry_pass
/opt/cloudera-manager/cm-5.16.1/share/cmf/schema/scm_prepare_database.sh mysql -h master -uroot -pcdh12#hadoop --scm-host % nav nav nav_pass
/opt/cloudera-manager/cm-5.16.1/share/cmf/schema/scm_prepare_database.sh mysql -h master -uroot -pcdh12#hadoop --scm-host % navms navms navms_pass
/opt/cloudera-manager/cm-5.16.1/share/cmf/schema/scm_prepare_database.sh mysql -h master -uroot -pcdh12#hadoop --scm-host % oozie oozie oozie_pass

    返回如下信息，表示配置成功

    [main] INFO  com.cloudera.enterprise.dbutil.DbCommandExecutor  - Successfully connected to database.
All done, your SCM database is configured correctly!

3.5 创建本地parcel源

  • 在master节点制作本地parcel源

    # 创建本地parcel源目录
mkdir -p /opt/cloudera/parcel-repo

# 将parcel相关安装包放置到"/opt/cloudera/parcel-repo"目录；
# 说明："/opt/cloudera/parcel-repo"目录可放置多套parcel安装包；
# 将"CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.sha1"重命名为"CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.sha"，否则会重新下载"CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel"安装包
mv /usr/local/src/CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel /opt/cloudera/parcel-repo/
mv /usr/local/src/CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.sha1 /opt/cloudera/parcel-repo/CDH-5.16.1-1.cdh5.16.1.p0.3-el7.parcel.sha
mv /usr/local/src/manifest.json /opt/cloudera/parcel-repo/

# 赋权
chown -R cloudera-scm:cloudera-scm /opt/cloudera/

  • 在所有salve节点创建软件安装目录

    mkdir -p /opt/cloudera/parcels

# 赋权
chown -R cloudera-scm:cloudera-scm /opt/cloudera/

3.6 启动CM服务

  • 在master节点启动cloudera-scm-server服务

    # "cloudera-scm-server"启动需要连接数据库，监听端口启动会延迟
service cloudera-scm-server restart
service cloudera-scm-server status -l

# 通过启动后的状态查看，脚本需要执行"pstree"命令，需要安装依赖包
yum install psmisc -y

  • 在所有salve节点启动cloudera-scm-agent服务

    yum install psmisc -y
service cloudera-scm-agent restart
service cloudera-scm-agent status -l

四. 安装CDH

4.1 CDH安装配置

  • 浏览器访问CM：http://172.30.200.75:7180
  • 默认账号/密码：admin/admin
  • 最终用户许可条款与条件
  • 部署版本，选择Cloudera Enterprise 试用版
  • 版本与服务说明
  • cloudera-scm-agent正常启动后，可发现相应的节点，指定集群服务的安装节点
  • 选择CDH版本
  • 集群安装，如果本地parcel源配置正确，则"下载"阶段瞬间完成，其余阶段视节点数与内部网络情况决定
  • 检查主机正确性(需要一些时间)，针对slave节点有两个优化建议：
    • 1.建议将/proc/sys/vm/swappiness设置为最大值10。
      • swappiness值控制操作系统尝试交换内存的积极；
      • swappiness=0：表示最大限度使用物理内存，之后才是swap空间；
      • swappiness=100：表示积极使用swap分区，并且把内存上的数据及时搬迁到swap空间；
      • 如果是混合服务器，不建议完全禁用swap，可以尝试降低swappiness。

      • 临时调整：

        sysctl vm.swappiness=10

      • 永久调整：

        cat << EOF >> /etc/sysctl.conf

# Adjust swappiness value
vm.swappiness=10
EOF
    • 2. Enabled transparent large page compression can cause significant performance issues, it is recommended to disable this setting.

      • Temporary adjustments:

        echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled

      • Permanent adjustments:

        << EOF >> /etc/rc.d/rc.local CAT 

# Disable transparent_hugepage 
echo Never> / SYS / Kernel / mm / transparent_hugepage / Defrag 
echo Never> / SYS / Kernel / mm / transparent_hugepage / Enabled 
EOF 

# centos7.x system, you need to "/etc/rc.d/rc.local" file gives execute permissions 
chmod + x /etc/rc.d/rc.local
  • Cluster setup, select the service to install, select here自定义服务的HDFS服务
  • Cluster setup, custom roles assigned by default to
  • Cluster setup,Reports Manager数据库名称rman，用户名rman
  • Cluster setup, review changes to default
  • Cluster setup, first run
  • Cluster setup, service installation is complete
  • Cloudera Manager Home
  • Cloudera Manager HDFS Service Summary

Guess you like

Origin www.linuxidc.com/Linux/2019-07/159618.htm
Recommended
Arc Browser for Windows 1.0 officially GA
A programmer born in the 1990s developed a video porting software and made over 7 million in less than a year. The ending was very punishing!
Ranking
1. Select Sort
Create a thread thread
3 press to play ball that reach 6
Programmation CUDA (4) : gestion de la mémoire
SpringBoot database connection pool Druid error
E Diudiu App redesign summary
4EVERLAND Hosting now supports SNS+IPFS
About HTTPS
[vue3+vite+ts+element-plu+sass] uses bug records in sass
Interpretation of HUAWEI CLOUD GaussDB (for Influx): Best Practice Data Modeling
Daily
More
2024-05-03(8)
2024-05-02(0)
2024-05-01(4)
2024-04-30(36)
2024-04-29(5)
2024-04-28(12)
2024-04-27(29)
2024-04-26(22)
2024-04-25(32)
2024-04-24(30)

Code World

© 2018 codetd.com