Our Cloudera Manager and cdh version is 5.14, the company now need to upgrade to cdh6.2
need to upgrade Cloudera Manager, then upgrade cdh.
1.Cloudera Manager upgrade
(Reference
https://www.cloudera.com/documentation/enterprise/upgrade/topics/ug_cm_upgrade.html )
to determine the linux version has been upgraded to support Cloudera Manager6.2 version before upgrading
Backup 1.1
1.1.1 Backup Cloudera Manager Agent
### view database information
$ cat /etc/cloudera-scm-server/db.propertiesGet the following information:
...
com.cloudera.cmf.db.type=...
com.cloudera.cmf.db.host=database_hostname:database_port
com.cloudera.cmf.db.name=scm
com.cloudera.cmf.db.user=scm
com.cloudera.cmf.db.password=SOME_PASSWORDIn each installation Cloudera Manager agent machines are to perform the following backup operations:
Create a top level backup directory.
$ export CM_BACKUP_DIR="`date +%F`-CM5.14"
$ echo $CM_BACKUP_DIR
$ mkdir -p $CM_BACKUP_DIRBack up the Agent directory and the runtime state.
$ sudo -E tar -cf $CM_BACKUP_DIR/cloudera-scm-agent.tar --exclude=*.sock /etc/cloudera-scm-agent /etc/default/cloudera-scm-agent /var/run/cloudera-scm-agent /var/lib/cloudera-scm-agentBack up the existing repository directory.
$ sudo -E tar -cf $CM_BACKUP_DIR/repository.tar /etc/yum.repos.d1.1.2 Backup Cloudera Manager Service
Executed on the Service Monitor installation of the machines:
$ sudo cp -rp /var/lib/cloudera-service-monitor /var/lib/cloudera-service-monitor-`date +%F`-CM5.14Host Monitor is executed on the installation of the machines:
$ sudo cp -rp /var/lib/cloudera-host-monitor /var/lib/cloudera-host-monitor-`date +%F`-CM5.14Performed on the computer where you installed the Event Server:
$ sudo cp -rp /var/lib/cloudera-scm-eventserver /var/lib/cloudera-scm-eventserver-`date +%F`-CM5.141.1.3 Backup Cloudera Manager Databases
$ mysqldump --databases database_name --host=database_hostname --port=database_port -u user_name -p > $HOME/database_name-backup-`date +%F`-CM5.14.sql1.1.2 Backup Cloudera Manager Server
Create a top-level backup directory.
export CM_BACKUP_DIR="`date +%F`-CM5.14"
echo $CM_BACKUP_DIR
mkdir -p $CM_BACKUP_DIRBack up the Cloudera Manager Server directories:
sudo -E tar -cf $CM_BACKUP_DIR/cloudera-scm-server.tar /etc/cloudera-scm-server /etc/default/cloudera-scm-serverBack up the existing repository directory.
sudo -E tar -cf $CM_BACKUP_DIR/repository.tar /etc/yum.repos.dCloudera Manager Server 1.2 upgrade
1.2.1 build software access rights (to replace yum source)
Log Cloudera Manager Server node, delete the original source yum
$ sudo rm /etc/yum.repos.d/cloudera*manager.repo* Create a new source file yum
[cloudera-manager]
# Packages for Cloudera Manager
name=Cloudera Manager
baseurl=https://archive.cloudera.com/cm6/6.2.0/redhat6/yum/
gpgkey=https://archive.cloudera.com/cm6/6.2.0/redhat6/yum/RPM-GPG-KEY-cloudera
gpgcheck=11.2.2 install or configure java8
Java_home configuration file in the configuration server:
in / etc / default / cloudera-scm -server
increase
export JAVA_HOME = "/ usr / java / jdk1.8.0_141-cloudera"
Cloudera Manager Server 1.2.3 upgrade
1. Log Cloudera Manager Server host.
2. Stop Cloudera management services. (Important: At this point do not stop Cloudera Management Service may result in administrative roles to crash or Cloudera Manager Server may not restart.)
Step:
- a.Log in to the Cloudera Manager Admin Console.
- b.Select Clusters > Cloudera Management Service.
- c.Select Actions > Stop.
-
3. Stop Cloudera Manager Server.
$ sudo service cloudera-scm-server stop4. Stop Cloudera Manager Agent.
$ sudo service cloudera-scm-agent stopThe upgrade Cloudera packages.
$ sudo yum clean all $ sudo yum upgrade cloudera-manager-server cloudera-manager-daemons cloudera-manager-agent6. Make sure that the package is installed under the
rpm -qa 'cloudera-manager-*'``` 7.启动Cloudera Manager Agent.sudo service cloudera-scm-agent start
8.启动Cloudera Manager Server.sudo service cloudera-scm-server start
If there are problems during startup can refer to a log file:
tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log
tail -f / var / log / Cloudera-SCM-Agent / Cloudera-scm- the agent.log
tail -f / var / log / messages
9. normal, then open cdh upgrade page you can see the situation the upgrade
http: // cloudera_Manager_server_hostname: 7180 / cmf / upgrade
1.2.4 upgrade Cloudera Manager Agent
. a CDH use interface upgrade
click Cloudera Manager Agent Package
Option 1: Select the agent repository
we use public libraries will be able to
choose Cloudera Repository Public
2. Install the JDK
already installed do not have to choose
3. Install the agent
configuration at root or sudo account number on it, we need access to all agent nodes permission
Option 2: Use the upgrade command
to clear the old repo file
sudo rm /etc/yum.repos.d/cloudera*manager.repo*New repo file:
vim /etc/yum.repos.d/cloudera-manager.reporepo file contents:
[cloudera-manager]
# Packages for Cloudera Manager
name=Cloudera Manager
baseurl=https://archive.cloudera.com/cm6/6.2.0/redhat6/yum/
gpgkey=https://archive.cloudera.com/cm6/6.2.0/redhat6/yum/RPM-GPG-KEY-cloudera
gpgcheck=1Stop Cloudera Manager agent service
sudo service cloudera-scm-agent stopUpgrade Cloudera Manager agent
sudo yum clean all
sudo yum repolist
sudo yum upgrade cloudera-manager-daemons cloudera-manager-agent So wait until after the machines are completed, each agent node perform
$ sudo service cloudera-scm-agent startView
http://192.168.0.254:7180/cmf/upgrade
show all agent machines have been upgraded, and has a heartbeat
click Host Inspector, detected at node case
Click to show results after completing the inspector to see the item in question, repair.
The problem appears to have a more important: If you want to run follow-up CDH6, hue need to use python2.7, first remember that no matter temporarily.
Then, start Cloudera Management Service
This completes the upgrade Cloudera Manager, and later upgrade cdh
If the upgrade fails, we need to restore, you can refer to the official steps:
https://www.cloudera.com/documentation/enterprise/upgrade/topics/ug_cm_downgrade.html
2.CDH upgrade
First determine before you upgrade the linux version has been upgraded to CDH6.2 supported version, java version 1.8
2.1. Preparations
Run the following command to check the cluster case
if there is a problem then repair
check hdfs:
$ sudo -u hdfs hdfs fsck / -includeSnapshots
$ sudo -u hdfs hdfs dfsadmin -reportConsistency check hbase table:
$ sudo -u HDFS hbase hbck
If the kudu, check kudu:
$ sudo -u kudu kudu cluster ksck <master_addresses>The following services 6.0.0 has not, before the upgrade, you need to stop and remove these services
Accumulo
Sqoop 2
MapReduce 1
the Spark 1.6
the Record Service
2.2 Backup cdh
CDH do not need to backup the following components:
· MapReduce
· YARN
· the Spark
· Pig
· Impala
Complete the following steps to back up before upgrading CDH
1.Back Up Databases
We use mysql, so to mysq for example
1) If it is not stopped, stop the service. If Cloudera Manager indicate the presence of dependent services, we have to stop the dependent services.
2) Backup individual services (Sqoop, Oozie, Hue, Hive Metastore, Sentry) database. Replace the database name, host name, port, username, and backup directory path, and then run the following command:
$ mysqldump --databases database_name --host = database_hostname --port = database_port -u database_username -p> backup_directory_path / database_name -backup -`date +％F`-CDH 5.14 .sql2.Back Up ZooKeeper
arranged in each node zookeeper, cdh the backup data storage directory zookeeper, such as
$ cp -rp /var/lib/zookeeper/ /var/lib/zookeeper-backup-`date +%F`CM-CDH5.14Up the HDFS 3.Back
(data path in accordance with a command cdh actual configuration change)
a. journal data backup is performed on each JournalNode
cp -rp /dfs/jn /dfs/jn-CM-CDH5.14. B Backup to run each time namenode directory, run:
$ mkdir -p /etc/hadoop/conf.rollback.namenode
$ cd /var/run/cloudera-scm-agent/process/ && cd `ls -t1 | grep -e "-NAMENODE\$" | head -1`
$ cp -rp * /etc/hadoop/conf.rollback.namenode/
$ rm -rf /etc/hadoop/conf.rollback.namenode/log4j.properties
$ cp -rp /etc/hadoop/conf.cloudera.HDFS_service_name/log4j.properties /etc/hadoop/conf.rollback.namenode/These commands create a temporary rollback directory. If you need to roll back to CDH 5.x later, the rollback process requires that you modify the files in this directory.
c. Each datanode backup runtime directory
$ mkdir -p /etc/hadoop/conf.rollback.datanode/
$ cd /var/run/cloudera-scm-agent/process/ && cd `ls -t1 | grep -e "-DATANODE\$" | head -1`
$ cp -rp * /etc/hadoop/conf.rollback.datanode/
$ rm -rf /etc/hadoop/conf.rollback.datanode/log4j.properties
$ cp -rp /etc/hadoop/conf.cloudera.HDFS_service_name/log4j.properties /etc/hadoop/conf.rollback.datanode/4.Back Up Key Trustee Server and Clients
Services does not use
5.Back Up HSM KMS
service does not use
6.Back Up Navigator Encrypt
service does not use
7.Back Up HBase
due to the rollback process will be rolled back HDFS, HBase data is therefore It will be rolled back. Further, HBase metadata stored in ZooKeeper will be restored as part of the rollback procedure ZooKeeper.
8.Back Up Search
service does not use
9.Back Up Sqoop 2
service does not use
10.Back Up Hue
in Hue Server running on all host role, the registry file backup app
$ mkdir -p /opt/cloudera/parcels_backup
$ cp -rp /opt/cloudera/parcels/CDH/lib/hue/app.reg /opt/cloudera/parcels_backup/app.reg-CM-CDH5.142.3 Service Changes:
hue：
For centos6 version of the system:
the need to install python2.7 node hue of
Enable the Software Collections Library:
$ sudo yum install centos-release-scl$ Install the Software Collections utilities:
$ sudo yum install scl-utils$ Install Python 2.7:
sudo yum install python27Verify that Python 2.7 is installed:
$ source /opt/rh/python27/enable
$ python --versionhbase:
1.HBase 2.0 does not support PREFIX_TREE block coding, you need to delete the first upgrade, otherwise hbase2.0 not start
CDH6 If you have installed it to ensure that all tables or snapshots do not use PREFIX_TREE block coding by running the following tools:
$ hbase pre-upgrade validate-dbe
$ hbase pre-upgrade validate-hfile2. Coprocessor class upgrade
external coprocessor will not be upgraded automatically. There are two ways to deal with co-processor upgrade:
Before proceeding with the upgrade manually upgrade coprocessor jar.
Suspended coprocessor settings and continue to escalate.
After manual upgrade, they can be reset.
Try to upgrade may result in unpredictable behavior, such as HBase role failed to start, HBase role collapse without upgrading coprocessor jar, or even data corruption.
If you have already installed CDH 6, you can be sure that your co-processor and upgrade compatible hbase pre-upgrade validate-cp by running the tool.
2.4 Upgrading the cluster
Precautions
When using Cloudera Manager Backup and Disaster Recovery (BDR ) to upgrade from a cluster Cloudera Manager 5.13 CDH or earlier to 6.0 or higher, using Cloudera Manager Backup and Disaster Recovery (BDR ) backup data will not work.
Performing a minor version upgrade of Cloudera Manager must be equal or greater than the minor version CDH. To upgrade Cloudera Manager
Note:
Use a rolling restart (only a minor upgrade) When you upgrade CDH:
automatic failover does not affect the rolling restart operations.
After the upgrade is complete, if it is running MapReduce jobs or Spark, do not delete the old block. These jobs are still using the old block, the block must be restarted to use the new upgrade.
Oozie ensure that work is idempotent.
Do not use Oozie Shell Actions to run Hadoop-related commands.
Does not support rolling upgrades Spark Streaming operations. Restart the job stream After the upgrade, in order to start using the new version deployed.
Runtime library must be packaged as part of the Spark application.
You must use a distributed cache propagation job configuration file from the client gateway computer.
Do not include third-party dependencies or build CDH class of "super" or "fat" JAR files because these files may, Oozie and other services and Yarn automatically added to the CLASSPATH of class conflict.
Spark build applications without bundling of CDH JAR.
2.4.1 Backup cloudera manager
Before cloudera manager upgrading our backup once, after the upgrade you need to be backed up.
1. Review the database information
$ cat /etc/cloudera-scm-server/db.propertiesE.g:
com.cloudera.cmf.db.type=...
com.cloudera.cmf.db.host=database_hostname:database_port
com.cloudera.cmf.db.name=scm
com.cloudera.cmf.db.user=scm
com.cloudera.cmf.db.password=SOME_PASSWORD2. Backup Cloudera Manager Agent
Execute on each agent node:
- Create a backup directory
$ export CM_BACKUP_DIR="`date +%F`-CM5.14" $ echo $CM_BACKUP_DIR $ mkdir -p $CM_BACKUP_DIR -
Backup agent directory and run-time status
$ sudo -E tar -cf $CM_BACKUP_DIR/cloudera-scm-agent.tar --exclude=*.sock /etc/cloudera-scm-agent /etc/default/cloudera-scm-agent /var/run/cloudera-scm-agent /var/lib/cloudera-scm-agent -
Back up the current repo directory
$ sudo -E tar -cf $ CM_BACKUP_DIR / repository.tar /etc/yum.repos.d -
Backup Cloudera Management Service
in Service Monitor node performs
$ sudo cp -rp / var / lib / Cloudera-Service-Monitor / var / lib / Cloudera-Service-Monitor - taking thedate +%F-CM5.14
executed on the Host Monitor node
$ sudo cp -rp / var / lib / Cloudera-Host-Monitor / var / lib / Monitor - taking the Cloudera-Host-date +%F-CM5.14
performed on the Event Server node$ sudo cp -rp /var/lib/cloudera-scm-eventserver /var/lib/cloudera-scm-eventserver-`date +%F`-CM5.143.停止Cloudera Manager Server & Cloudera Management Service
Stop in CDH administration interface Cloudera Management Service, select:
. Clusters-> Cloudera Management Service
. Actions> Stop
Stop Cloudera Manager Server:$ sudo service cloudera-scm-server stop4. Cloudera Manager database backup
$ mysqldump --databases database_name --host=database_hostname --port=database_port -u user_name -p > $HOME/database_name-backup-`date +%F`-CM5.14.sqlInformation database to obtain information for just the first step to view the file
5. Back up Cloudera Manager Server
In Cloudera Manager Server node to perform:
1. Create a backup directory:$ export CM_BACKUP_DIR="`date +%F`-CM5.14" $ echo $CM_BACKUP_DIR $ mkdir -p $CM_BACKUP_DIR2. Contents of Cloudera Manager Server Backup
$ Tar -cf sudo -E $ CM_BACKUP_DIR / Cloudera-scm-server.tar / etc / Cloudera-scm-Server / etc / default / Cloudera-scm-Server
3. Backup the current repo directory$ sudo -E tar -cf $CM_BACKUP_DIR/repository.tar /etc/yum.repos.d
2.4.2 goes into maintenance mode
要在升级过程中避免不必要的警报，请在开始升级之前在群集上进入维护模式。 进入维护模式会停止发送电子邮件警报和SNMP陷阱，但不会停止检查和配置验证。完成升级后，请务必退出维护模式以重新启用Cloudera Manager警报。2.4.3 upgrade completed before the migration steps
- the Yarn
Decommission and recommission at The YARN NodeManagers But do not Start at The NodeManagers.
A Decommission IS required SO that at The NodeManagers STOP accepting new new Containers, the kill the any running Containers, and the then the shutdown. (The YARN of NodeManagers to Decommission out, then recommission, but not start NodeManagers, need to Decommission NodeManagers stop accepting new container, container terminate all running, and then close.)Steps:
1. Make sure before the upgrade is complete, no new applications (such as Spark or MapReduce applications) submitted to the cluster.
2. Open the CDH management interface, enter the service you want to upgrade YARN
3. On the Instances tab, select all NodeManager roles. This can be done by filtering role in the types of roles
4. Click on the selected operation -> Decommissioning
If the cluster is running CDH 5.9 or later by Cloudera Manager 5.9 or later manage, and you have configured properly decommissioned it will start the countdown timeout.
It provides a smooth retirement before the start of the process to stop using a timeout. Timeout creates a window of time in order to consume the workload has been running from the system, and allow them to run to completion. On the Configuration tab YARN Service Node Manager Graceful Decommission Timeout search field, and set the property value greater than 0 to create a timeout.
5. Wait until complete decommissioned. Upon completion, NodeManager status is stopped, the authorization status for the lifting of the authorization.
6. Check all NodeManagers, click on the selected operation -> reauthorization.
Important: Do not start select all NodeManager.
- hive
Query syntax, DDL syntax and Hive API has changed. Before you upgrade, you may need to edit HiveQL code in your application workloads.
-
sentry
If your cluster uses Sentry authorization policy file, you must migrate first policy document to support the Sentry database service, and then upgrade to CDH 6. - spark
If the cluster using Spark or Spark Standalone, you must perform several steps to ensure that the correct version.
Delete spark standalone
After the upgrade, if installed spark2, the spark2-submit is replaced with spark-submit, to submit job command before submitting jobs to replace
2.4.4 Hue document clean-up operation
If the cluster using the Hue, perform the following steps (maintenance release is not required). Hue these steps to clean up the database tables used, can help improve the performance after the upgrade.
1. Hue database backup.
2. Connect to Hue database.
3. Check desktop_document, desktop_document2, oozie_job, beeswax_session, beeswax_savedquery and size beeswax_queryhistory table to obtain the reference point. If any row over 100,000 lines, run the cleanup.
2.4.5 download and distribute parcels
1. Open the CDH management interface, click on the host -> Parcels -> Configuration
2. Use the following remote parcel repository URL of the update of CDH Parcel repository:
https://archive.cloudera.com/cdh6/6.2.0/parcels/
a. In a remote part of the URL Parcel repository, single "+" icon to add the above url, click Save Changes
b. Find the line containing the new CDH parcel in the table, then click the "Download" button.
c. After downloading the package, click the "Assign" button.
d. When finished distributing all packages, click on the Upgrade button.
2.4.6 run the Upgrade Wizard CDH
1. After entering the Upgrade Wizard, will run some cluster check, check results may be some problems, it will affect subsequent upgrade, first to solve these problems. There will be prompted to back up the database. If you have been ok, click Yes, I have performed these steps, and then click Continue.
2. Click on the full cluster reboot (all downtime cluster), click Continue. (This step will restart all services)
The upgrade process encountered some problems:
The upgrade process exception in Oozie shared library:
1. "java.lang.ClassNotFoundException: org.cloudera.log4j.redactor.RedactorAppender" class not found.
Refer to the article, the lack of logredactor-2.0.7.jar build a soft link from / opt / cloudera / parcels / CDH / lib / oozie / lib to / opt / cloudera / parcels / CDH / lib / oozie / libtools directory under
2.ERROR StatusLogger No log4j2 configuration file found Using default configuration:.. Logging only errors to the console
because log4j.xml not lead to abnormal configuration information can not be displayed, the same test a log4j.xml templates into / opt / cloudera / parcels / CDH / lib / oozie / under libtools directory.
2.4.7 After the completion of the migration upgrade
1.spark
- After upgrading to CDH 6, may be configured with multiple Spark service, each service has its own set of configurations, including the event log position. Determine which service you want to keep, and then manually merge the two services.
-
Spark 2 commit command for the job (spark2-submit) in CDH 5 deleted in the CDH 6,
replace spark-submit. Spark 1.6 with a built-in services and service CDH 5 Spark 2 clusters, used in conjunction with spark-submit service Spark 1.6 and spark2-submit service used together with Spark 2. After upgrading to CDH 6, spark-submit CDH built using service Spark 2, spark2-submit no longer work. Be sure to use these commands to update any submission workflow Spark job. - By performing the following steps to manually merge Spark Service:
1. All configuration is copied from the service to delete the service you want to keep. To view and edit the configuration:
. A In Cloudera Manager Admin Console, go to the Spark service you want to delete.
b. Click the "Configure" tab.
C. Configuration Record.
d. Go Spark service you want to keep and replication configuration.
e. Click Save changes.
: To preserve the history of the event log
location you sure you want to delete the service event log:
In Cloudera Manager Admin Console, go to delete Spark service.
Click the "Configure" tab.
Search: spark.eventLog.dir
note path.
Log on to the cluster host and run the following command:
hadoop FS -mv <old_Spark_Event_Log_dir> / * <new_location> /.
Use Cloudera Manager, stop and delete the service you choose to remove the Spark
restart the remaining Spark service: Click the drop-down arrow next to the service Spark, then select "Restart."2.impala
impala mainly used even if the query is not used for online tasks, so the importance of not so high, refer to the official website
https://www.cloudera.com/documentation/enterprise/upgrade/topics/impala_upgrading.html
can be.