安装部署CDH6.3.2

企业开发 2023-12-17 06:40:06 阅读次数: 0

安装部署CDH

Cloudera Manager

概述

CDH (Cloudera Distribution Including Apache Hadoop) 是一个基于 Apache Hadoop 的开源软件发行版。它是由Cloudera公司开发和维护的，为企业和组织提供了易于使用和管理的大数据处理解决方案。

CDH包含了一系列与Hadoop生态系统相关的开源软件，如Apache Hadoop、Apache Spark、Apache HBase、Apache Hive、Apache Impala、Apache Flume、Apache Sqoop等等。这些软件用于存储、处理和分析大规模数据集。

Cloudera为CDH提供了增强的集成、管理和安全功能，使得企业可以更轻松地部署和操作Hadoop群集。CDH还提供了一些工具和界面，用于监视和优化群集性能，以及支持各种数据处理工作负载。

主要功能

集群部署和配置：

Cloudera Manager简化了CDH集群的部署和配置过程。它提供了预定义的部署模板、角色分配和配置向导，使管理员可以快速配置各个组件和服务。

监控和警报：

Cloudera Manager监控整个大数据集群的运行状况和性能指标。它收集和展示有关各个节点、服务和作业的关键指标，并提供实时和历史性能图表。此外，它还能生成警报以提醒管理员关于潜在的问题或重要事件。

故障排除和诊断：

Cloudera Manager提供了故障排除和诊断工具，帮助管理员定位和解决集群中的问题。它提供了错误和异常日志的集中查看、问题诊断工具和故障排除向导，以加快故障修复的过程。

自动化任务和操作：

Cloudera Manager使管理员能够执行自动化任务和操作，如软件安装、配置更改、备份和恢复等。它提供了一致的集群操作和管理界面，使管理员可以在整个集群上进行一致的操作。

架构

Cloudera Manager的架构实现了集中式管理和监控的能力，管理员可以通过Web界面和API对整个集群进行配置、部署、监控和管理。

Cloudera Manager Server：

Cloudera Manager的核心组件，负责管理整个集群和与用户进行交互。它提供了一个Web界面和API，用于集群配置、部署、监控和管理。

Agents：

在每个集群节点上安装并在Cloudera Manager Server指导下运行的代理程序。它们与Cloudera Manager Server通信，收集节点的运行状况和性能指标，并执行命令和任务。

数据库：

使用数据库存储集群和配置元数据。可以选择使用内置的Derby数据库或外部数据库，如MySQL或PostgreSQL。

Cloudera Management Services：

执行各种管理任务的服务，包括Activity Monitor、Alert Publisher、Event Server、Reports Manager、Service Monitor和Host Monitor。

环境准备

节点情况

准备5个节点

vim /etc/hosts

172.29.234.1 node01
172.29.234.2 node02
172.29.234.3 node03
172.29.234.4 node04
172.29.234.5 node05

SSH免密配置

配置各个节点进行免密登录

参考：SSH免密配置

安装JDK

在各个节点安装JDK

参考：安装JDK

安装MySQL

选取一台节点安装MySQL

参考：安装MySQL

创建数据库

CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

CM安装部署

上传cdh安装文件

创建/opt/cloudera-manager目录，存放cdh安装文件

tar -zxvf cm6.3.1-redhat7.tar.gz

cd cm6.3.1/RPMS/x86_64/

mkdir /opt/cloudera-manager

mv cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm /opt/cloudera-manager
mv cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm /opt/cloudera-manager/
mv cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm /opt/cloudera-manager/

 cd /opt/cloudera-manager/

分发安装文件到各个节点

[root@node01 ~]# ./sync.sh /opt/cloudera-manager/

安装cloudera-manager-daemons

各个节点执行安装cloudera-manager-daemons

[root@node01 ~]# ./call.sh rpm -ivh /opt/cloudera-manager/cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm
--------- node01 ----------
warning: /opt/cloudera-manager/cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY
Preparing...                          ########################################      
--------- node02 ----------
warning: /opt/cloudera-manager/cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY
Preparing...                          ########################################      
--------- node03 ----------
warning: /opt/cloudera-manager/cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY
Preparing...                          ########################################     
--------- node04 ----------
warning: /opt/cloudera-manager/cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY
Preparing...                          ######################################## 
--------- node05 ----------
warning: /opt/cloudera-manager/cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY
Preparing...                          ########################################

安装完毕后会多出一个/opt/cloudera目录

[root@node01 cloudera-manager]# ls /opt/
cloudera

安装cloudera-manager-agent

各个节点安装cloudera-manager-agent

安装cloudera-manager-agent时，需要安装相关依赖，否则将出现如下异常：

[root@node01 ~]# ./call.sh rpm -ivh /opt/cloudera-manager/cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm
--------- node01 ----------
warning: /opt/cloudera-manager/cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY
error: Failed dependencies:
        bind-utils is needed by cloudera-manager-agent-6.3.1-1466458.el7.x86_64
        libxslt is needed by cloudera-manager-agent-6.3.1-1466458.el7.x86_64
        cyrus-sasl-plain is needed by cloudera-manager-agent-6.3.1-1466458.el7.x86_64
        cyrus-sasl-gssapi is needed by cloudera-manager-agent-6.3.1-1466458.el7.x86_64
        /lib/lsb/init-functions is needed by cloudera-manager-agent-6.3.1-1466458.el7.x86_64
        httpd is needed by cloudera-manager-agent-6.3.1-1466458.el7.x86_64
        mod_ssl is needed by cloudera-manager-agent-6.3.1-1466458.el7.x86_64
        openssl-devel is needed by cloudera-manager-agent-6.3.1-1466458.el7.x86_64
        python-psycopg2 is needed by cloudera-manager-agent-6.3.1-1466458.el7.x86_64
        MySQL-python is needed by cloudera-manager-agent-6.3.1-1466458.el7.x86_64
        libpq.so.5()(64bit) is needed by cloudera-manager-agent-6.3.1-1466458.el7.x86_64

解决方案则是安装如下依赖：

yum install bind-utils psmisc cyrus-sasl-plain cyrus-sasl-gssapi fuse portmap fuse-libs /lib/lsb/init-functions httpd mod_ssl openssl-devel python-psycopg2 MySQL-python libxslt

安装依赖时，发现在已安装了MySQL的节点出现如下异常：

--> Running transaction check
---> Package libsepol-devel.x86_64 0:2.5-10.el7 will be installed
---> Package pcre-devel.x86_64 0:8.32-17.el7 will be installed
Removing mariadb-libs.x86_64 1:5.5.68-1.el7 - u due to obsoletes from installed mysql-community-libs-5.7.43-1.el7.x86_64
--> Restarting Dependency Resolution with new changes.
--> Running transaction check
---> Package mariadb-libs.x86_64 1:5.5.68-1.el7 will be installed
--> Processing Dependency: libmysqlclient.so.18(libmysqlclient_18)(64bit) for package: MySQL-python-1.2.5-1.el7.x86_64
--> Processing Dependency: libmysqlclient.so.18()(64bit) for package: MySQL-python-1.2.5-1.el7.x86_64
--> Finished Dependency Resolution
Error: Package: MySQL-python-1.2.5-1.el7.x86_64 (base)
           Requires: libmysqlclient.so.18()(64bit)
Error: Package: MySQL-python-1.2.5-1.el7.x86_64 (base)
           Requires: libmysqlclient.so.18(libmysqlclient_18)(64bit)
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest

原因：

在mysql5.7版本中，缺失libmysqlclient.so.18()(64bit)这个依赖，因为安装MySQL时没有安装

解决方案:

安装mysql-community-libs-compat-5.7.43-1.el7.x86_64.rpm

[root@node01 ~]# rpm -ivh mysql-lib/mysql-community-libs-compat-5.7.43-1.el7.x86_64.rpm
warning: mysql-lib/mysql-community-libs-compat-5.7.43-1.el7.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID 3a79bd29: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:mysql-community-libs-compat-5.7.4################################# [100%]

使用脚本批量安装相关依赖：

[root@node01 ~]# ./call.sh  yum -y install bind-utils libxslt psmisc cyrus-sasl-plain cyrus-sasl-gssapi fuse portmap fuse-libs /lib/lsb/init-functions httpd mod_ssl openssl-devel python-psycopg2 MySQL-python

使用脚本批量安装agent：

[root@node01 ~]# ./call.sh rpm -ivh /opt/cloudera-manager/cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm                                                                             --------- node01 ----------
warning: /opt/cloudera-manager/cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY
Preparing...                          ########################################
Updating / installing...
cloudera-manager-agent-6.3.1-1466458.e########################################
Created symlink from /etc/systemd/system/multi-user.target.wants/cloudera-scm-agent.service to /usr/lib/systemd/system/cloudera-scm-agent.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/supervisord.service to /usr/lib/systemd/system/supervisord.service.
--------- node02 ----------
warning: /opt/cloudera-manager/cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY
Preparing...                          ########################################
Updating / installing...
cloudera-manager-agent-6.3.1-1466458.e########################################
Created symlink from /etc/systemd/system/multi-user.target.wants/cloudera-scm-agent.service to /usr/lib/systemd/system/cloudera-scm-agent.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/supervisord.service to /usr/lib/systemd/system/supervisord.service.
--------- node03 ----------
warning: /opt/cloudera-manager/cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY
Preparing...                          ########################################
Updating / installing...
cloudera-manager-agent-6.3.1-1466458.e########################################
Created symlink from /etc/systemd/system/multi-user.target.wants/cloudera-scm-agent.service to /usr/lib/systemd/system/cloudera-scm-agent.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/supervisord.service to /usr/lib/systemd/system/supervisord.service.
--------- node04 ----------
warning: /opt/cloudera-manager/cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY
Preparing...                          ########################################
Updating / installing...
cloudera-manager-agent-6.3.1-1466458.e########################################
Created symlink from /etc/systemd/system/multi-user.target.wants/cloudera-scm-agent.service to /usr/lib/systemd/system/cloudera-scm-agent.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/supervisord.service to /usr/lib/systemd/system/supervisord.service.
--------- node05 ----------
warning: /opt/cloudera-manager/cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY
Preparing...                          ########################################
Updating / installing...
cloudera-manager-agent-6.3.1-1466458.e########################################
Created symlink from /etc/systemd/system/multi-user.target.wants/cloudera-scm-agent.service to /usr/lib/systemd/system/cloudera-scm-agent.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/supervisord.service to /usr/lib/systemd/system/supervisord.service.

修改vim /etc/cloudera-scm-agent/config.ini文件，配置Agent，告知Server节点

[General]
# Hostname of the CM server.
#server_host=localhost
server_host=node01

# Port that the CM server is listening on.
server_port=7182

分发配置到各个节点

[root@node01 ~]# ./sync.sh /etc/cloudera-scm-agent/config.ini

安装cloudera-manager-server

只在node01节点安装cloudera-manager-server

[root@node01 ~]# rpm -ivh /opt/cloudera-manager/cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm
warning: /opt/cloudera-manager/cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:cloudera-manager-server-6.3.1-146################################# [100%]
Created symlink from /etc/systemd/system/multi-user.target.wants/cloudera-scm-server.service to /usr/lib/systemd/system/cloudera-scm-server.service.

上传CDH包到parcel-repo

[root@node01 parcel-repo]# mv CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel* /opt/cloudera/parcel-repo

[root@node01 parcel-repo]# mv manifest.json /opt/cloudera/parcel-repo/

[root@node01 parcel-repo]# cd /opt/cloudera/parcel-repo

[root@node01 parcel-repo]# mv CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1 CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha

[root@node01 parcel-repo]# ls
CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel  CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha  manifest.json

修改server的db.properties

vim /etc/cloudera-scm-server/db.properties

# Copyright (c) 2012 Cloudera, Inc. All rights reserved.
#
# This file describes the database connection.
#

# The database type
# Currently 'mysql', 'postgresql' and 'oracle' are valid databases.
com.cloudera.cmf.db.type=mysql

# The database host
# If a non standard port is needed, use 'hostname:port'
#com.cloudera.cmf.db.host=localhost
com.cloudera.cmf.db.host=node01:3306

# The database name
#com.cloudera.cmf.db.name=cmf
com.cloudera.cmf.db.name=scm

# The database user
#com.cloudera.cmf.db.user=cmf
com.cloudera.cmf.db.user=root

# The database user's password
#com.cloudera.cmf.db.password=
com.cloudera.cmf.db.password=123456

# The db setup type
# After fresh install it is set to INIT
# and will be changed post config.
# If scm-server uses Embedded DB then it is set to EMBEDDED
# If scm-server uses External DB then it is set to EXTERNAL

# com.cloudera.cmf.db.setupType=INIT
com.cloudera.cmf.db.setupType=EXTERNAL

导入数据

cm_prepare_database.sh是Cloudera Manager的脚本，用于准备数据库以供Cloudera Manage 使用。

具体执行命令如下：

/opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm root

参数mysql：指示使用 MySQL 数据库

参数scm：是数据库名称

参数root：是数据库用户名

当导入数据时，出现如下错误：

[root@node01 ~]# /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm root
Enter SCM password:
JAVA_HOME=/usr/local/jdk8
Verifying that we can write to /etc/cloudera-scm-server
Creating SCM configuration file in /etc/cloudera-scm-server
Executing:  /usr/local/jdk8/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera/cm/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
log4j:ERROR Could not find value for key log4j.appender.A
log4j:ERROR Could not instantiate appender named "A".
[2023-08-15 20:08:11,878] INFO     0[main] - com.cloudera.enterprise.dbutil.DbCommandExecutor.loadJDBCDriver(DbCommandExecutor.java) - Unable to find JDBC driver for database type: MySQL
[2023-08-15 20:08:11,881]ERROR     3[main] - com.cloudera.enterprise.dbutil.DbCommandExecutor$DbConnectionTestException.logError(DbCommandExecutor.java) - JDBC Driver com.mysql.jdbc.Driver not found.
[2023-08-15 20:08:11,881]ERROR     3[main] - com.cloudera.enterprise.dbutil.DbCommandExecutor$DbConnectionTestException.logError(DbCommandExecutor.java) - Exiting with exit code 3
--> Error 3, giving up (use --force if you wish to ignore the error)

解决方案：

下载mysql-connector-java-5.1.49.jar，将其复制到/usr/share/java路径下。

注意：改名为上述异常提到的名称：mysql-connector-java.jar

mv mysql-connector-java-5.1.49.jar  mysql-connector-java.jar

mkdir /usr/share/java

mv mysql-connector-java.jar /usr/share/java/

再次执行，操作成功如下：

[root@node01 ~]# /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm root
Enter SCM password:
JAVA_HOME=/usr/local/jdk8
Verifying that we can write to /etc/cloudera-scm-server
Creating SCM configuration file in /etc/cloudera-scm-server
Executing:  /usr/local/jdk8/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera/cm/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
log4j:ERROR Could not find value for key log4j.appender.A
log4j:ERROR Could not instantiate appender named "A".
Tue Aug 15 20:18:40 CST 2023 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
[2023-08-15 20:18:40,951] INFO     0[main] - com.cloudera.enterprise.dbutil.DbCommandExecutor.testDbConnection(DbCommandExecutor.java) - Successfully connected to database.
All done, your SCM database is configured correctly!

启动server服务

启动server服务

systemctl start cloudera-scm-server

停止server服务

systemctl stop cloudera-scm-server

查看server服务状态

[root@node01 ~]# systemctl status cloudera-scm-server
● cloudera-scm-server.service - Cloudera CM Server Service
   Loaded: loaded (/usr/lib/systemd/system/cloudera-scm-server.service; enabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Tue 2023-08-15 20:46:56 CST; 1min 47s ago
  Process: 328 ExecStart=/opt/cloudera/cm/bin/cm-server (code=exited, status=1/FAILURE)
  Process: 325 ExecStartPre=/opt/cloudera/cm/bin/cm-server-pre (code=exited, status=0/SUCCESS)
 Main PID: 328 (code=exited, status=1/FAILURE)

Aug 15 20:46:56 node01 systemd[1]: cloudera-scm-server.service: main process exited, code=exited, status=1/FAILURE
Aug 15 20:46:56 node01 systemd[1]: Unit cloudera-scm-server.service entered failed state.
Aug 15 20:46:56 node01 systemd[1]: cloudera-scm-server.service failed.
Aug 15 20:46:56 node01 systemd[1]: cloudera-scm-server.service holdoff time over, scheduling restart.
Aug 15 20:46:56 node01 systemd[1]: Stopped Cloudera CM Server Service.
Aug 15 20:46:56 node01 systemd[1]: start request repeated too quickly for cloudera-scm-server.service
Aug 15 20:46:56 node01 systemd[1]: Failed to start Cloudera CM Server Service.
Aug 15 20:46:56 node01 systemd[1]: Unit cloudera-scm-server.service entered failed state.
Aug 15 20:46:56 node01 systemd[1]: cloudera-scm-server.service failed.

启动失败，执行journalctl -xe命令查看具体异常信息

[root@node01 ~]# journalctl -xe

Aug 15 20:46:56 node01 cm-server[328]: +======================================================================+
Aug 15 20:46:56 node01 cm-server[328]: |      Error: Unable to find a compatible version of Java on this host,|
Aug 15 20:46:56 node01 cm-server[328]: |             either because JAVA_HOME has not been set or because a   |
Aug 15 20:46:56 node01 cm-server[328]: |             compatible version of Java is not installed.             |
Aug 15 20:46:56 node01 cm-server[328]: +----------------------------------------------------------------------+
Aug 15 20:46:56 node01 cm-server[328]: | Please install either:                                               |
Aug 15 20:46:56 node01 cm-server[328]: | - a supported version of the Oracle JDK from the Oracle Java web     |
Aug 15 20:46:56 node01 cm-server[328]: |   site:                                                              |
Aug 15 20:46:56 node01 cm-server[328]: |   > http://www.oracle.com/technetwork/java/javase/index.html <       |
Aug 15 20:46:56 node01 cm-server[328]: | OR                                                                   |
Aug 15 20:46:56 node01 cm-server[328]: | - a supported version of the OpenJDK from your OS vendor.  Help for  |
Aug 15 20:46:56 node01 cm-server[328]: |   some OSes are available at:                                        |
Aug 15 20:46:56 node01 cm-server[328]: |   > http://openjdk.java.net/install/ <                               |
Aug 15 20:46:56 node01 cm-server[328]: |                                                                      |
Aug 15 20:46:56 node01 cm-server[328]: | Cloudera Manager requires Oracle JDK or OpenJDK 1.8 or later.        |
Aug 15 20:46:56 node01 cm-server[328]: | NOTE: Cloudera Manager will find the Oracle JDK when starting,       |
Aug 15 20:46:56 node01 cm-server[328]: |       regardless of whether you installed the JDK using a binary     |
Aug 15 20:46:56 node01 cm-server[328]: |       installer or the RPM-based installer.                          |
Aug 15 20:46:56 node01 cm-server[328]: +======================================================================+
Aug 15 20:46:56 node01 systemd[1]: cloudera-scm-server.service: main process exited, code=exited, status=1/FAILURE
Aug 15 20:46:56 node01 systemd[1]: Unit cloudera-scm-server.service entered failed state.
Aug 15 20:46:56 node01 systemd[1]: cloudera-scm-server.service failed.
Aug 15 20:46:56 node01 systemd[1]: cloudera-scm-server.service holdoff time over, scheduling restart.
Aug 15 20:46:56 node01 systemd[1]: Stopped Cloudera CM Server Service.

根据异常信息提示，找不到JAVA_HOME，喊安装JDK，明明已经安装及配置好了JAVA_HOME。

具体原因：Server会使用/usr/java下的jdk。因此，解决方案：

mkdir -p /usr/java

ln -s /usr/local/jdk8/  /usr/java/default

重新启动

[root@node01 ~]# systemctl status cloudera-scm-server
● cloudera-scm-server.service - Cloudera CM Server Service
   Loaded: loaded (/usr/lib/systemd/system/cloudera-scm-server.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2023-08-15 21:08:04 CST; 5s ago
  Process: 8626 ExecStartPre=/opt/cloudera/cm/bin/cm-server-pre (code=exited, status=0/SUCCESS)
 Main PID: 8629 (java)
   CGroup: /system.slice/cloudera-scm-server.service
           └─8629 /usr/java/default/bin/java -cp .:/usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:lib/* -serv...

Aug 15 21:08:04 node01 systemd[1]: Starting Cloudera CM Server Service...
Aug 15 21:08:04 node01 systemd[1]: Started Cloudera CM Server Service.
Aug 15 21:08:04 node01 cm-server[8629]: JAVA_HOME=/usr/java/default

查看日志

tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log

当进度达到100%，出现7180端口号，就可以访问Web界面

2023-08-15 21:09:37,671 INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: SPNEGO authentication is not enabled
2023-08-15 21:09:37,704 INFO WebServerImpl:com.cloudera.server.web.cmon.JobDetailGatekeeper: ActivityMonitor configured to allow job details for all jobs.
2023-08-15 21:09:38,060 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   ... 40% complete
2023-08-15 21:09:39,197 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   ... 50% complete
2023-08-15 21:09:40,132 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   ... 60% complete
2023-08-15 21:09:40,791 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Initializing SearchTemplateManager:2023-08-15T13:09:40.791Z
2023-08-15 21:09:40,842 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Generating entities:2023-08-15T13:09:40.842Z
2023-08-15 21:09:40,861 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Num entities:223
2023-08-15 21:09:40,861 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Generating documents:2023-08-15T13:09:40.861Z
2023-08-15 21:09:40,925 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Num docs:236
2023-08-15 21:09:40,926 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Constructing repo:2023-08-15T13:09:40.925Z
2023-08-15 21:09:41,528 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   ... 70% complete
2023-08-15 21:09:42,344 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Finished constructing repo:2023-08-15T13:09:42.344Z
2023-08-15 21:09:42,708 INFO WebServerImpl:org.eclipse.jetty.server.Server: jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_371-b11
2023-08-15 21:09:42,728 INFO WebServerImpl:org.eclipse.jetty.server.AbstractConnector: Started ServerConnector@7b124d81{
    
    HTTP/1.1,[http/1.1]}{
    
    0.0.0.0:7180}
2023-08-15 21:09:42,728 INFO WebServerImpl:org.eclipse.jetty.server.Server: Started @97631ms
2023-08-15 21:09:42,728 INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.
2023-08-15 21:09:42,951 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   ... 80% complete
2023-08-15 21:09:43,448 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   ... 90% complete
2023-08-15 21:09:44,010 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: Hashed 1 file(s) (2082186246 bytes) in 3972 pieces (3972 expected) in 11334.4ms.
2023-08-15 21:09:44,015 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: Single-file torrent information:
2023-08-15 21:09:44,015 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   Torrent name: CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel
2023-08-15 21:09:44,015 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   Announced at: Seems to be trackerless
2023-08-15 21:09:44,015 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   Created on..: Tue Aug 15 21:09:32 CST 2023
2023-08-15 21:09:44,015 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   Created by..: cm-server
2023-08-15 21:09:44,015 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   Pieces......: 3972 piece(s) (524288 byte(s)/piece)
2023-08-15 21:09:44,015 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   Total size..: 2,082,186,246 byte(s)

启动agent节点

[root@node01 ~]# ./call.sh systemctl start cloudera-scm-agent
--------- node01 ----------
--------- node02 ----------
--------- node03 ----------
--------- node04 ----------
--------- node05 ----------

停止agent节点

./call.sh systemctl stop cloudera-scm-agent

CM的使用

访问CM

浏览器访问：http://IP:7180，使用用户名：admin 密码：admin进行登录

在这里插入图片描述

接收条款与选择订阅类型

接受最终用户许可条款和条件

在这里插入图片描述
选择使用免费许可类型
在这里插入图片描述

集群安装

设置集群名称
在这里插入图片描述
勾选所有节点主机
在这里插入图片描述
选择CDH版本，默认即可
在这里插入图片描述
等待下载安装
在这里插入图片描述
注意：

在重新安装时，可能遇到主机运行状态不良的情况

原因：Agent的集群id与当前Server集群id不匹配造成的

解决方案：删除agent的cm_guid文件

[root@node01 ~]# ./call.sh systemctl stop cloudera-scm-agent

[root@node01 ~]# find / -name cm_guid
/var/lib/cloudera-scm-agent/cm_guid

[root@node01 ~]# cat /var/lib/cloudera-scm-agent/cm_guid
4b751103-b1fc-4914-81d1-52b2ebc20913

[root@node01 ~]# ./call.sh rm -rf /var/lib/cloudera-scm-agent/cm_guid

[root@node01 ~]# ./call.sh systemctl start cloudera-scm-agent

点击检查网络性能与检查主机
在这里插入图片描述
在这里插入图片描述
在这里插入图片描述

第二个异常，按照提示处理即可：

[root@node01 ~]# ./call.sh "echo never > /sys/kernel/mm/transparent_hugepage/defrag"
[root@node01 ~]# ./call.sh "echo never > /sys/kernel/mm/transparent_hugepage/enabled"

第一个与第三个异常信息不明显，点击查看具体详细信息
在这里插入图片描述
大概意思是Java版本不兼容，解决办法就是更换受支持的Oracle JDK版本
在这里插入图片描述

rpm -ivh oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm
warning: oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:oracle-j2sdk1.8-1.8.0+update181-1################################# [100%]
[root@node01 ~]# ./call.sh /usr/java/jdk1.8.0_181-cloudera/


[root@node01 ~]# whereis java
java: /usr/share/java /usr/java/jdk1.8.0_181-cloudera/bin/java

配置环境变量：vim /etc/bashrc

export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib
export PATH=$PATH:$JAVA_HOME/bin

验证安装

[root@node01 ~]# java -version
java version "1.8.0_371"
Java(TM) SE Runtime Environment (build 1.8.0_371-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.371-b11, mixed mode)

同步分发文件、配置

[root@node01 ~]# ./sync.sh /usr/java/jdk1.8.0_181-cloudera/

[root@node01 ~]# ./sync.sh /etc/bashrc

[root@node01 ~]# ./call.sh source /etc/bashrc

重新启动Server与Agent

systemctl start cloudera-scm-server

./call.sh systemctl start cloudera-scm-agent

再次查看，发现异常已消失，仅剩一个异常，按照上述第二个异常处理即可
在这里插入图片描述
在这里插入图片描述

群集设置

选择自定义安装
在这里插入图片描述
选择安装服务，这里选择安装HDFS、YARN、ZK等三个基础服务
在这里插入图片描述
分配节点
在这里插入图片描述
集群设置全部选默认即可
在这里插入图片描述
开始自动安装
在这里插入图片描述
等待安装后，其会自动启动进程
在这里插入图片描述

首页

在这里插入图片描述

配置NameNode HA

进入HDFS页面点击启用High Availability
在这里插入图片描述
在这里插入图片描述
分配角色
在这里插入图片描述
审核更改
在这里插入图片描述
等待启动服务
在这里插入图片描述
在这里插入图片描述

配置Yarn HA

在这里插入图片描述
在这里插入图片描述
在这里插入图片描述

在这里插入图片描述

猜你喜欢

转载自blog.csdn.net/qq_38628046/article/details/132299440
今日推荐
国产云输入法——仅华为无云端数据上传安全问题
开源日报 | 工业开源项目OGG 1.0；姐姐，你要和我一起配置火狐吗；苹果AI遥遥落后？Fedora 40
开放签电子签章：停止新增，优化体验，前进更进（五一假期前工作）
开源日报 | 中学生开源前端动画引擎；全球首个Llama3 8B中文版开源模型；联想电脑恐出局；Linus讽刺AI炒作
“百模大战”必有一战 | 2024中国“百模大战”竞争格局分析
最强开源大模型 Llama 3 上架 Gitee AI
周排行
自媒体文章如何提高原创度以及如何检测原创度
开启qq邮箱的smtp服务
Qt程序单次启动（QSingleApplication类）
国外的外包网站
更新IDEA主题——放飞代码风格
cocos2dx 实现搓牌效果（翻牌效果），包括铺平动画
dict和json之间的互相转换
angular的一些思考
. Fibonacci数列是这样定义的： F[0] = 0 F[1] = 1 for each i ≥ 2: F[i] = F[i-1] + F[i-2] 因此，Fibonacci数列就形如：0, 1
洛谷P1064 金明的预算方案
每日归档
更多
2024-04-25(22)
2024-04-24(36)
2024-04-23(26)
2024-04-22(39)
2024-04-21(0)
2024-04-20(6)
2024-04-19(5)
2024-04-18(0)
2024-04-17(5)
2024-04-16(70)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022