Install and deploy CDH6.3.2

Enterprise 2023-12-17 07:40:38 views: null

Install and deploy CDH

Cloudera Manager

Overview

CDH (Cloudera Distribution Including Apache Hadoop) is an open source software distribution based on Apache Hadoop. Developed and maintained by Cloudera, it provides enterprises and organizations with easy-to-use and manage big data processing solutions.

CDH contains a series of open source software related to the Hadoop ecosystem, such as Apache Hadoop, Apache Spark, Apache HBase, Apache Hive, Apache Impala, Apache Flume, Apache Sqoop, etc. These software are used to store, process, and analyze large-scale data sets.

Cloudera provides enhanced integration, management and security features for CDH, making it easier for enterprises to deploy and operate Hadoop clusters. CDH also provides tools and interfaces for monitoring and optimizing cluster performance, as well as supporting a variety of data processing workloads.

The main function

Cluster deployment and configuration:

Cloudera Manager simplifies the deployment and configuration process of CDH clusters. It provides predefined deployment templates, role assignments, and configuration wizards that allow administrators to quickly configure various components and services.

Monitoring and alerting:

Cloudera Manager monitors the health and performance metrics of the entire big data cluster. It collects and displays key metrics about individual nodes, services, and jobs, and provides real-time and historical performance graphs. In addition, it can generate alerts to alert administrators about potential problems or important events.

Troubleshooting and Diagnostics:

Cloudera Manager provides troubleshooting and diagnostic tools to help administrators locate and solve problems in the cluster. It provides centralized viewing of error and exception logs, problem diagnosis tools, and troubleshooting wizards to speed up the process of troubleshooting.

Automate tasks and operations:

Cloudera Manager enables administrators to perform automated tasks and operations such as software installation, configuration changes, backup and recovery, etc. It provides a consistent cluster operation and management interface, allowing administrators to perform consistent operations across the entire cluster.

Architecture

The architecture of Cloudera Manager enables centralized management and monitoring capabilities. Administrators can configure, deploy, monitor and manage the entire cluster through the web interface and API.

Cloudera Manager Server：

The core component of Cloudera Manager, responsible for managing the entire cluster and interacting with users. It provides a web interface and API for cluster configuration, deployment, monitoring and management.

Agents：

An agent installed on each cluster node and run under the direction of Cloudera Manager Server. They communicate with Cloudera Manager Server, collect node health and performance metrics, and execute commands and tasks.

database:

Use a database to store cluster and configuration metadata. You can choose to use the built-in Derby database or an external database such as MySQL or PostgreSQL.

Cloudera Management Services：

Services that perform various management tasks, including Activity Monitor, Alert Publisher, Event Server, Reports Manager, Service Monitor, and Host Monitor.

Environmental preparation

Node status

Prepare 5 nodes

vim /etc/hosts

172.29.234.1 node01
172.29.234.2 node02
172.29.234.3 node03
172.29.234.4 node04
172.29.234.5 node05

SSH password-free configuration

Configure each node for password-free login

Reference:SSH immune deployment

Install JDK

Install JDK on each node

Reference:Anso JDK

InstallMySQL

Select a node to install MySQL

Reference:Anso MySQL

Create database

CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;

CM installation and deployment

Upload cdh installation files

Create/opt/cloudera-manager directory to store cdh installation files

tar -zxvf cm6.3.1-redhat7.tar.gz

cd cm6.3.1/RPMS/x86_64/

mkdir /opt/cloudera-manager

mv cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm /opt/cloudera-manager
mv cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm /opt/cloudera-manager/
mv cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm /opt/cloudera-manager/

 cd /opt/cloudera-manager/

Distribute installation files to each node

[root@node01 ~]# ./sync.sh /opt/cloudera-manager/

Install cloudera-manager-daemons

Each node executes the installation of cloudera-manager-daemons

[root@node01 ~]# ./call.sh rpm -ivh /opt/cloudera-manager/cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm
--------- node01 ----------
warning: /opt/cloudera-manager/cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY
Preparing...                          ########################################      
--------- node02 ----------
warning: /opt/cloudera-manager/cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY
Preparing...                          ########################################      
--------- node03 ----------
warning: /opt/cloudera-manager/cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY
Preparing...                          ########################################     
--------- node04 ----------
warning: /opt/cloudera-manager/cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY
Preparing...                          ######################################## 
--------- node05 ----------
warning: /opt/cloudera-manager/cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY
Preparing...                          ########################################

After the installation is completed, there will be one more directory/opt/clouderadirectory

[root@node01 cloudera-manager]# ls /opt/
cloudera

Install cloudera-manager-agent

Install cloudera-manager-agent on each node

When installingcloudera-manager-agent, you need to install relevant dependencies, otherwise the following exception will occur:

[root@node01 ~]# ./call.sh rpm -ivh /opt/cloudera-manager/cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm
--------- node01 ----------
warning: /opt/cloudera-manager/cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY
error: Failed dependencies:
        bind-utils is needed by cloudera-manager-agent-6.3.1-1466458.el7.x86_64
        libxslt is needed by cloudera-manager-agent-6.3.1-1466458.el7.x86_64
        cyrus-sasl-plain is needed by cloudera-manager-agent-6.3.1-1466458.el7.x86_64
        cyrus-sasl-gssapi is needed by cloudera-manager-agent-6.3.1-1466458.el7.x86_64
        /lib/lsb/init-functions is needed by cloudera-manager-agent-6.3.1-1466458.el7.x86_64
        httpd is needed by cloudera-manager-agent-6.3.1-1466458.el7.x86_64
        mod_ssl is needed by cloudera-manager-agent-6.3.1-1466458.el7.x86_64
        openssl-devel is needed by cloudera-manager-agent-6.3.1-1466458.el7.x86_64
        python-psycopg2 is needed by cloudera-manager-agent-6.3.1-1466458.el7.x86_64
        MySQL-python is needed by cloudera-manager-agent-6.3.1-1466458.el7.x86_64
        libpq.so.5()(64bit) is needed by cloudera-manager-agent-6.3.1-1466458.el7.x86_64

The solution is to install the following dependencies:

yum install bind-utils psmisc cyrus-sasl-plain cyrus-sasl-gssapi fuse portmap fuse-libs /lib/lsb/init-functions httpd mod_ssl openssl-devel python-psycopg2 MySQL-python libxslt

When installing dependencies, I found the following exception on the node where MySQL is installed:

--> Running transaction check
---> Package libsepol-devel.x86_64 0:2.5-10.el7 will be installed
---> Package pcre-devel.x86_64 0:8.32-17.el7 will be installed
Removing mariadb-libs.x86_64 1:5.5.68-1.el7 - u due to obsoletes from installed mysql-community-libs-5.7.43-1.el7.x86_64
--> Restarting Dependency Resolution with new changes.
--> Running transaction check
---> Package mariadb-libs.x86_64 1:5.5.68-1.el7 will be installed
--> Processing Dependency: libmysqlclient.so.18(libmysqlclient_18)(64bit) for package: MySQL-python-1.2.5-1.el7.x86_64
--> Processing Dependency: libmysqlclient.so.18()(64bit) for package: MySQL-python-1.2.5-1.el7.x86_64
--> Finished Dependency Resolution
Error: Package: MySQL-python-1.2.5-1.el7.x86_64 (base)
           Requires: libmysqlclient.so.18()(64bit)
Error: Package: MySQL-python-1.2.5-1.el7.x86_64 (base)
           Requires: libmysqlclient.so.18(libmysqlclient_18)(64bit)
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest

reason:

In version mysql5.7, this dependency is missing libmysqlclient.so.18()(64bit) because it was not installed when installing MySQL

solution:

Installmysql-community-libs-compat-5.7.43-1.el7.x86_64.rpm

[root@node01 ~]# rpm -ivh mysql-lib/mysql-community-libs-compat-5.7.43-1.el7.x86_64.rpm
warning: mysql-lib/mysql-community-libs-compat-5.7.43-1.el7.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID 3a79bd29: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:mysql-community-libs-compat-5.7.4################################# [100%]

Use script to batch install related dependencies:

[root@node01 ~]# ./call.sh  yum -y install bind-utils libxslt psmisc cyrus-sasl-plain cyrus-sasl-gssapi fuse portmap fuse-libs /lib/lsb/init-functions httpd mod_ssl openssl-devel python-psycopg2 MySQL-python

Use script to install agents in batches:

[root@node01 ~]# ./call.sh rpm -ivh /opt/cloudera-manager/cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm                                                                             --------- node01 ----------
warning: /opt/cloudera-manager/cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY
Preparing...                          ########################################
Updating / installing...
cloudera-manager-agent-6.3.1-1466458.e########################################
Created symlink from /etc/systemd/system/multi-user.target.wants/cloudera-scm-agent.service to /usr/lib/systemd/system/cloudera-scm-agent.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/supervisord.service to /usr/lib/systemd/system/supervisord.service.
--------- node02 ----------
warning: /opt/cloudera-manager/cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY
Preparing...                          ########################################
Updating / installing...
cloudera-manager-agent-6.3.1-1466458.e########################################
Created symlink from /etc/systemd/system/multi-user.target.wants/cloudera-scm-agent.service to /usr/lib/systemd/system/cloudera-scm-agent.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/supervisord.service to /usr/lib/systemd/system/supervisord.service.
--------- node03 ----------
warning: /opt/cloudera-manager/cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY
Preparing...                          ########################################
Updating / installing...
cloudera-manager-agent-6.3.1-1466458.e########################################
Created symlink from /etc/systemd/system/multi-user.target.wants/cloudera-scm-agent.service to /usr/lib/systemd/system/cloudera-scm-agent.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/supervisord.service to /usr/lib/systemd/system/supervisord.service.
--------- node04 ----------
warning: /opt/cloudera-manager/cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY
Preparing...                          ########################################
Updating / installing...
cloudera-manager-agent-6.3.1-1466458.e########################################
Created symlink from /etc/systemd/system/multi-user.target.wants/cloudera-scm-agent.service to /usr/lib/systemd/system/cloudera-scm-agent.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/supervisord.service to /usr/lib/systemd/system/supervisord.service.
--------- node05 ----------
warning: /opt/cloudera-manager/cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY
Preparing...                          ########################################
Updating / installing...
cloudera-manager-agent-6.3.1-1466458.e########################################
Created symlink from /etc/systemd/system/multi-user.target.wants/cloudera-scm-agent.service to /usr/lib/systemd/system/cloudera-scm-agent.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/supervisord.service to /usr/lib/systemd/system/supervisord.service.

Modifyvim /etc/cloudera-scm-agent/config.ini file, configure Agent, notify Server node

[General]
# Hostname of the CM server.
#server_host=localhost
server_host=node01

# Port that the CM server is listening on.
server_port=7182

Distribute configuration to each node

[root@node01 ~]# ./sync.sh /etc/cloudera-scm-agent/config.ini

Install cloudera-manager-server

Only install cloudera-manager-server on the node01 node

[root@node01 ~]# rpm -ivh /opt/cloudera-manager/cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm
warning: /opt/cloudera-manager/cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:cloudera-manager-server-6.3.1-146################################# [100%]
Created symlink from /etc/systemd/system/multi-user.target.wants/cloudera-scm-server.service to /usr/lib/systemd/system/cloudera-scm-server.service.

Upload CDH package to parcel-repo

[root@node01 parcel-repo]# mv CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel* /opt/cloudera/parcel-repo

[root@node01 parcel-repo]# mv manifest.json /opt/cloudera/parcel-repo/

[root@node01 parcel-repo]# cd /opt/cloudera/parcel-repo

[root@node01 parcel-repo]# mv CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1 CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha

[root@node01 parcel-repo]# ls
CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel  CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha  manifest.json

Modify the server's db.properties

vim /etc/cloudera-scm-server/db.properties

# Copyright (c) 2012 Cloudera, Inc. All rights reserved.
#
# This file describes the database connection.
#

# The database type
# Currently 'mysql', 'postgresql' and 'oracle' are valid databases.
com.cloudera.cmf.db.type=mysql

# The database host
# If a non standard port is needed, use 'hostname:port'
#com.cloudera.cmf.db.host=localhost
com.cloudera.cmf.db.host=node01:3306

# The database name
#com.cloudera.cmf.db.name=cmf
com.cloudera.cmf.db.name=scm

# The database user
#com.cloudera.cmf.db.user=cmf
com.cloudera.cmf.db.user=root

# The database user's password
#com.cloudera.cmf.db.password=
com.cloudera.cmf.db.password=123456

# The db setup type
# After fresh install it is set to INIT
# and will be changed post config.
# If scm-server uses Embedded DB then it is set to EMBEDDED
# If scm-server uses External DB then it is set to EXTERNAL

# com.cloudera.cmf.db.setupType=INIT
com.cloudera.cmf.db.setupType=EXTERNAL

Import Data

cm_prepare_database.shIs a script for Cloudera Manager that prepares the database for use by Cloudera Manage.

The specific execution commands are as follows:

/opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm root

参数mysql：指示使用 MySQL 数据库

参数scm：是数据库名称

参数root：是数据库用户名

When importing data, the following error occurs:

[root@node01 ~]# /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm root
Enter SCM password:
JAVA_HOME=/usr/local/jdk8
Verifying that we can write to /etc/cloudera-scm-server
Creating SCM configuration file in /etc/cloudera-scm-server
Executing:  /usr/local/jdk8/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera/cm/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
log4j:ERROR Could not find value for key log4j.appender.A
log4j:ERROR Could not instantiate appender named "A".
[2023-08-15 20:08:11,878] INFO     0[main] - com.cloudera.enterprise.dbutil.DbCommandExecutor.loadJDBCDriver(DbCommandExecutor.java) - Unable to find JDBC driver for database type: MySQL
[2023-08-15 20:08:11,881]ERROR     3[main] - com.cloudera.enterprise.dbutil.DbCommandExecutor$DbConnectionTestException.logError(DbCommandExecutor.java) - JDBC Driver com.mysql.jdbc.Driver not found.
[2023-08-15 20:08:11,881]ERROR     3[main] - com.cloudera.enterprise.dbutil.DbCommandExecutor$DbConnectionTestException.logError(DbCommandExecutor.java) - Exiting with exit code 3
--> Error 3, giving up (use --force if you wish to ignore the error)

solution:

Downloadmysql-connector-java-5.1.49.jar and copy it to the/usr/share/java path.

NOTE: Rename to the name mentioned in the above exception:mysql-connector-java.jar

mv mysql-connector-java-5.1.49.jar  mysql-connector-java.jar

mkdir /usr/share/java

mv mysql-connector-java.jar /usr/share/java/

Executed again, the operation is successful as follows:

[root@node01 ~]# /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm root
Enter SCM password:
JAVA_HOME=/usr/local/jdk8
Verifying that we can write to /etc/cloudera-scm-server
Creating SCM configuration file in /etc/cloudera-scm-server
Executing:  /usr/local/jdk8/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera/cm/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
log4j:ERROR Could not find value for key log4j.appender.A
log4j:ERROR Could not instantiate appender named "A".
Tue Aug 15 20:18:40 CST 2023 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
[2023-08-15 20:18:40,951] INFO     0[main] - com.cloudera.enterprise.dbutil.DbCommandExecutor.testDbConnection(DbCommandExecutor.java) - Successfully connected to database.
All done, your SCM database is configured correctly!

Start the server service

Start the server service

systemctl start cloudera-scm-server

Stop server service

systemctl stop cloudera-scm-server

Check server service status

[root@node01 ~]# systemctl status cloudera-scm-server
● cloudera-scm-server.service - Cloudera CM Server Service
   Loaded: loaded (/usr/lib/systemd/system/cloudera-scm-server.service; enabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Tue 2023-08-15 20:46:56 CST; 1min 47s ago
  Process: 328 ExecStart=/opt/cloudera/cm/bin/cm-server (code=exited, status=1/FAILURE)
  Process: 325 ExecStartPre=/opt/cloudera/cm/bin/cm-server-pre (code=exited, status=0/SUCCESS)
 Main PID: 328 (code=exited, status=1/FAILURE)

Aug 15 20:46:56 node01 systemd[1]: cloudera-scm-server.service: main process exited, code=exited, status=1/FAILURE
Aug 15 20:46:56 node01 systemd[1]: Unit cloudera-scm-server.service entered failed state.
Aug 15 20:46:56 node01 systemd[1]: cloudera-scm-server.service failed.
Aug 15 20:46:56 node01 systemd[1]: cloudera-scm-server.service holdoff time over, scheduling restart.
Aug 15 20:46:56 node01 systemd[1]: Stopped Cloudera CM Server Service.
Aug 15 20:46:56 node01 systemd[1]: start request repeated too quickly for cloudera-scm-server.service
Aug 15 20:46:56 node01 systemd[1]: Failed to start Cloudera CM Server Service.
Aug 15 20:46:56 node01 systemd[1]: Unit cloudera-scm-server.service entered failed state.
Aug 15 20:46:56 node01 systemd[1]: cloudera-scm-server.service failed.

Startup failed, executejournalctl -xe command to view specific exception information

[root@node01 ~]# journalctl -xe

Aug 15 20:46:56 node01 cm-server[328]: +======================================================================+
Aug 15 20:46:56 node01 cm-server[328]: |      Error: Unable to find a compatible version of Java on this host,|
Aug 15 20:46:56 node01 cm-server[328]: |             either because JAVA_HOME has not been set or because a   |
Aug 15 20:46:56 node01 cm-server[328]: |             compatible version of Java is not installed.             |
Aug 15 20:46:56 node01 cm-server[328]: +----------------------------------------------------------------------+
Aug 15 20:46:56 node01 cm-server[328]: | Please install either:                                               |
Aug 15 20:46:56 node01 cm-server[328]: | - a supported version of the Oracle JDK from the Oracle Java web     |
Aug 15 20:46:56 node01 cm-server[328]: |   site:                                                              |
Aug 15 20:46:56 node01 cm-server[328]: |   > http://www.oracle.com/technetwork/java/javase/index.html <       |
Aug 15 20:46:56 node01 cm-server[328]: | OR                                                                   |
Aug 15 20:46:56 node01 cm-server[328]: | - a supported version of the OpenJDK from your OS vendor.  Help for  |
Aug 15 20:46:56 node01 cm-server[328]: |   some OSes are available at:                                        |
Aug 15 20:46:56 node01 cm-server[328]: |   > http://openjdk.java.net/install/ <                               |
Aug 15 20:46:56 node01 cm-server[328]: |                                                                      |
Aug 15 20:46:56 node01 cm-server[328]: | Cloudera Manager requires Oracle JDK or OpenJDK 1.8 or later.        |
Aug 15 20:46:56 node01 cm-server[328]: | NOTE: Cloudera Manager will find the Oracle JDK when starting,       |
Aug 15 20:46:56 node01 cm-server[328]: |       regardless of whether you installed the JDK using a binary     |
Aug 15 20:46:56 node01 cm-server[328]: |       installer or the RPM-based installer.                          |
Aug 15 20:46:56 node01 cm-server[328]: +======================================================================+
Aug 15 20:46:56 node01 systemd[1]: cloudera-scm-server.service: main process exited, code=exited, status=1/FAILURE
Aug 15 20:46:56 node01 systemd[1]: Unit cloudera-scm-server.service entered failed state.
Aug 15 20:46:56 node01 systemd[1]: cloudera-scm-server.service failed.
Aug 15 20:46:56 node01 systemd[1]: cloudera-scm-server.service holdoff time over, scheduling restart.
Aug 15 20:46:56 node01 systemd[1]: Stopped Cloudera CM Server Service.

According to the exception message, JAVA_HOME cannot be found, and it is called to install JDK. JAVA_HOME has obviously been installed and configured.

Specific reason: Server will use jdk under/usr/java. So, the solution:

mkdir -p /usr/java

ln -s /usr/local/jdk8/  /usr/java/default

Restart

[root@node01 ~]# systemctl status cloudera-scm-server
● cloudera-scm-server.service - Cloudera CM Server Service
   Loaded: loaded (/usr/lib/systemd/system/cloudera-scm-server.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2023-08-15 21:08:04 CST; 5s ago
  Process: 8626 ExecStartPre=/opt/cloudera/cm/bin/cm-server-pre (code=exited, status=0/SUCCESS)
 Main PID: 8629 (java)
   CGroup: /system.slice/cloudera-scm-server.service
           └─8629 /usr/java/default/bin/java -cp .:/usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:lib/* -serv...

Aug 15 21:08:04 node01 systemd[1]: Starting Cloudera CM Server Service...
Aug 15 21:08:04 node01 systemd[1]: Started Cloudera CM Server Service.
Aug 15 21:08:04 node01 cm-server[8629]: JAVA_HOME=/usr/java/default

View log

tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log

When the progress reaches 100%, the port number appears7180, and you can access the web interface

2023-08-15 21:09:37,671 INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: SPNEGO authentication is not enabled
2023-08-15 21:09:37,704 INFO WebServerImpl:com.cloudera.server.web.cmon.JobDetailGatekeeper: ActivityMonitor configured to allow job details for all jobs.
2023-08-15 21:09:38,060 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   ... 40% complete
2023-08-15 21:09:39,197 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   ... 50% complete
2023-08-15 21:09:40,132 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   ... 60% complete
2023-08-15 21:09:40,791 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Initializing SearchTemplateManager:2023-08-15T13:09:40.791Z
2023-08-15 21:09:40,842 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Generating entities:2023-08-15T13:09:40.842Z
2023-08-15 21:09:40,861 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Num entities:223
2023-08-15 21:09:40,861 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Generating documents:2023-08-15T13:09:40.861Z
2023-08-15 21:09:40,925 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Num docs:236
2023-08-15 21:09:40,926 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Constructing repo:2023-08-15T13:09:40.925Z
2023-08-15 21:09:41,528 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   ... 70% complete
2023-08-15 21:09:42,344 INFO SearchRepositoryManager-0:com.cloudera.server.web.cmf.search.components.SearchRepositoryManager: Finished constructing repo:2023-08-15T13:09:42.344Z
2023-08-15 21:09:42,708 INFO WebServerImpl:org.eclipse.jetty.server.Server: jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 1.8.0_371-b11
2023-08-15 21:09:42,728 INFO WebServerImpl:org.eclipse.jetty.server.AbstractConnector: Started ServerConnector@7b124d81{
    
    HTTP/1.1,[http/1.1]}{
    
    0.0.0.0:7180}
2023-08-15 21:09:42,728 INFO WebServerImpl:org.eclipse.jetty.server.Server: Started @97631ms
2023-08-15 21:09:42,728 INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.
2023-08-15 21:09:42,951 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   ... 80% complete
2023-08-15 21:09:43,448 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   ... 90% complete
2023-08-15 21:09:44,010 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: Hashed 1 file(s) (2082186246 bytes) in 3972 pieces (3972 expected) in 11334.4ms.
2023-08-15 21:09:44,015 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent: Single-file torrent information:
2023-08-15 21:09:44,015 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   Torrent name: CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel
2023-08-15 21:09:44,015 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   Announced at: Seems to be trackerless
2023-08-15 21:09:44,015 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   Created on..: Tue Aug 15 21:09:32 CST 2023
2023-08-15 21:09:44,015 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   Created by..: cm-server
2023-08-15 21:09:44,015 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   Pieces......: 3972 piece(s) (524288 byte(s)/piece)
2023-08-15 21:09:44,015 INFO ParcelUpdateService:com.turn.ttorrent.common.Torrent:   Total size..: 2,082,186,246 byte(s)

Start agent node

[root@node01 ~]# ./call.sh systemctl start cloudera-scm-agent
--------- node01 ----------
--------- node02 ----------
--------- node03 ----------
--------- node04 ----------
--------- node05 ----------

Stop agent node

./call.sh systemctl stop cloudera-scm-agent

Use of CM

Visit CM

Browser access:http://IP:7180, use username: admin and password: admin to log in

Insert image description here

Accept terms and choose subscription type

Accept the End User License Terms and Conditions

Insert image description here
Choose to use a free license type
Insert image description here

Cluster installation

Set the cluster name
Insert image description here
Check all node hosts
Insert image description here
Select the CDH version, the default is
Insert image description here
Wait for download and installation
Insert image description here
Note: 

在重新安装时，可能遇到主机运行状态不良的情况

原因：Agent的集群id与当前Server集群id不匹配造成的

解决方案：删除agent的cm_guid文件

[root@node01 ~]# ./call.sh systemctl stop cloudera-scm-agent

[root@node01 ~]# find / -name cm_guid
/var/lib/cloudera-scm-agent/cm_guid

[root@node01 ~]# cat /var/lib/cloudera-scm-agent/cm_guid
4b751103-b1fc-4914-81d1-52b2ebc20913

[root@node01 ~]# ./call.sh rm -rf /var/lib/cloudera-scm-agent/cm_guid

[root@node01 ~]# ./call.sh systemctl start cloudera-scm-agent

Click Check Network Performance and Check Host
Insert image description here
Insert image description here
Insert image description here

For the second exception, just follow the prompts to handle it:

[root@node01 ~]# ./call.sh "echo never > /sys/kernel/mm/transparent_hugepage/defrag"
[root@node01 ~]# ./call.sh "echo never > /sys/kernel/mm/transparent_hugepage/enabled"

The first and third exception information is not obvious, click to view the specific details
Insert image description here
It probably means that the Java version is incompatible. The solution is to replace the supported Oracle JDK version< /span>
Insert image description here

rpm -ivh oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm
warning: oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID b0b19c9f: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...
   1:oracle-j2sdk1.8-1.8.0+update181-1################################# [100%]
[root@node01 ~]# ./call.sh /usr/java/jdk1.8.0_181-cloudera/


[root@node01 ~]# whereis java
java: /usr/share/java /usr/java/jdk1.8.0_181-cloudera/bin/java

Configure environment variables:vim /etc/bashrc

export JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera
export CLASSPATH=.:$CLASSPATH:$JAVA_HOME/lib
export PATH=$PATH:$JAVA_HOME/bin

Verify installation

[root@node01 ~]# java -version
java version "1.8.0_371"
Java(TM) SE Runtime Environment (build 1.8.0_371-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.371-b11, mixed mode)

Synchronize distribution files and configurations

[root@node01 ~]# ./sync.sh /usr/java/jdk1.8.0_181-cloudera/

[root@node01 ~]# ./sync.sh /etc/bashrc

[root@node01 ~]# ./call.sh source /etc/bashrc

Restart Server and Agent

systemctl start cloudera-scm-server

./call.sh systemctl start cloudera-scm-agent

Check again and find that the exception has disappeared and there is only one exception left. Just handle it according to the second exception above.
Insert image description here
Insert image description here

Cluster settings

Select custom installation
Insert image description here
Select the installation service, here select three basic services including installation Allocate nodes Just select the default for all cluster settings Start automatic installation After waiting for the installation, it will automatically start the processHDFS、YARN、ZK
Insert image description here

Insert image description here

Insert image description here

Insert image description here

Insert image description here

front page

Insert image description here

Placement NameNode HA

Enter the HDFS page and click Enable High Availability
Insert image description here
Insert image description here
Assign roles
Insert image description here
Review changes
Insert image description here
Wait to start the service
Insert image description here
Insert image description here

Configure Yarn HA

Insert image description here
Insert image description here
Insert image description here

Insert image description here

Guess you like

Origin blog.csdn.net/qq_38628046/article/details/132299440
Recommended
Ranking
A quick primer on numpy basics
Two sister tease python project
Java | Multiple keywords in the replacement content are returned to the front end in red style
C ++: references
The string leetcood 1018 is a binary prefix divisible by 5
Flutter 布局组件
Java combat spingboot-redis
PMP pre-exam sprint and wrong questions sorting
ActiveMq C# Message Features: Delayed and Timed Message Delivery
DSP28377S_ copy a program from the RAM to FLASH portion DETAILS
Daily
More
2024-05-16(23)
2024-05-15(5)
2024-05-14(9)
2024-05-13(8)
2024-05-12(28)
2024-05-11(32)
2024-05-10(34)
2024-05-09(32)
2024-05-08(18)
2024-05-07(34)

Code World

© 2018 codetd.com