oracle19c rac patch upgrade

Preface

This article introduces the detailed process of upgrading RU (upgrading to 19.19) from Oracle 19C (19.17).

1. Introduction to RU

        Oracle releases one RU each quarter in January, April, July and October every year. To enable customers to get the latest security-related fixes as soon as their environment stabilizes, RURs will be provided for the two most recent RUs in January, April, July, October every year.

        RU (Release Updates) is a quarterly version upgrade. Including: the latest BUG fixes, the latest security vulnerability patches, and the latest added features. Taking 19c as an example, the corresponding upgrade version upgrade route is (19.1.0.0.0->19.2.0.0.0->19.3.0.0.0....).

        RUR is a revised version of RU, including: the latest BUG fixes and the latest security vulnerability patches. The latest features are not added here. And there are only revisions corresponding to the RU version. For example, in July 2020, in addition to releasing RU (19.8.0.0.0), last quarter's .1 patch (19.7.1.0) and last quarter's .2 patch (19.6.2.0) will be released.

2. Current environment

2.1. Patch details

Create a working directory and upload the patch to this directory for subsequent upgrades.

mkdir -p /backup/upgrade
chown grid:oinstall -R /backup/upgrade

Patch 35037840 - GI Release Update 19.19.0.0.230418
Patch 35042068 - Database Release Update 19.19.0.0.230418

2.2. Operating system information

[oracle@rac01:/home/oracle]$ cat /etc/os-release 
NAME="Oracle Linux Server"
VERSION="7.6"
ID="ol"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.6"
PRETTY_NAME="Oracle Linux Server 7.6"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:oracle:linux:7:6:server"
HOME_URL="https://linux.oracle.com/"
BUG_REPORT_URL="https://bugzilla.oracle.com/"

ORACLE_BUGZILLA_PRODUCT="Oracle Linux 7"
ORACLE_BUGZILLA_PRODUCT_VERSION=7.6
ORACLE_SUPPORT_PRODUCT="Oracle Linux"
ORACLE_SUPPORT_PRODUCT_VERSION=7.6

2.3. Database information

[oracle@rac01:/home/oracle]$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Fri Jul 21 14:30:39 2023
Version 19.17.0.0.0

Copyright (c) 1982, 2022, Oracle.  All rights reserved.


Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.17.0.0.0

SQL> set linesize 300
SQL> select comp_id,status,version from dba_registry;

COMP_ID                        STATUS                                       VERSION
------------------------------ -------------------------------------------- ------------------------------
CATALOG                        VALID                                        19.0.0.0.0
CATPROC                        VALID                                        19.0.0.0.0
RAC                            VALID                                        19.0.0.0.0
JAVAVM                         VALID                                        19.0.0.0.0
XML                            VALID                                        19.0.0.0.0
CATJAVA                        VALID                                        19.0.0.0.0
APS                            VALID                                        19.0.0.0.0
XDB                            VALID                                        19.0.0.0.0
OWM                            VALID                                        19.0.0.0.0
CONTEXT                        VALID                                        19.0.0.0.0
ORDIM                          VALID                                        19.0.0.0.0

COMP_ID                        STATUS                                       VERSION
------------------------------ -------------------------------------------- ------------------------------
SDO                            VALID                                        19.0.0.0.0
XOQ                            VALID                                        19.0.0.0.0
OLS                            VALID                                        19.0.0.0.0
DV                             VALID                                        19.0.0.0.0

15 rows selected.

2.3. Installed patch information

# grid用户通过如下命令收集当前GI补丁信息：
$ORACLE_HOME/OPatch/opatch lsinventory > /tmp/grid_patch.txt
$ORACLE_HOME/OPatch/opatch lsinv -detail -oh $ORACLE_HOME >> /tmp/grid_patch.txt
$ORACLE_HOME/OPatch/opatch lspatches >> /tmp/grid_patch.txt

# oracle用户通过如下命令收集当前DB补丁信息：
$ORACLE_HOME/OPatch/opatch lsinventory > /tmp/oracle_patch.txt
$ORACLE_HOME/OPatch/opatch lsinv -detail -oh $ORACLE_HOME >> /tmp/oracle_patch.txt
$ORACLE_HOME/OPatch/opatch lspatches >> /tmp/oracle_patch.txt

2.4.  Current status of DB components

Get component information, invalid object information:

sqlplus / as sysdba
spool /tmp/database_invalid.sql
set linesize 300
select comp_id,status,version from dba_registry;
select owner,object_type,object_name from dba_objects where status<>'VALID';
select PATCH_ID,ACTION,STATUS,ACTION_TIME from dba_registry_sqlpatch order by ACTION_TIME; 
spool off；

2.5.  Current status of cluster resources

The grid user obtains the cluster process and resource status:

crsctl check crs
crsctl check cluster -all
crsctl status res -t

2.6.  Monitoring service status

 Each grid user node obtains:

lsnrctl status

3. Backup before upgrading

        Because the patch installation is aimed at replacing the files in the Oracle installation directory, it is necessary to make a backup of the installation directory files. In order to ensure that the backed-up files are available, it is recommended to stop the database before backing up. When using a RAC cluster, it is recommended to stop the clusterware backup. When an unresolved problem occurs, the backed-up ORACLE_HOME directory can be restored and the clusterware can be started normally.

        When backing up, it is recommended to use the following command as the root user to back up with permissions. After the backup is completed, start the cluster and database.

# 备份前，使用grid用户停止所有database
srvctl stop database -d orcl
# 备份前，使用root用户停止所有集群服务
crsctl stop cluster -all
# 使用root用户带权限打包备份
# 所有节点分别备份
tar -czvpf u01_bak.tar.gz /u01/app/

# 启动集群
crsctl start cluster -all
# 启动数据库
srvctl start database -d orcl

4. OPatch upgrade

4.1.  Current version information

Get the current opatch version information of grid and oracle:

$ORACLE_HOME/OPatch/opatch version

4.2.New  version update

        If the current opatch version information does not meet the current patching requirements (you can check the readme in the patch set to be updated to confirm the opatch version requirements), you need to download the latest opatch package that conforms to the current database version from the mos website to replace the old version of the opatch directory. If it is lost, both GI and DB need to be replaced and upgraded.

su - grid
cd $ORACLE_HOME
mv OPatch OPatch_bak
cd /backup/upgrade
# 解压后注意权限
unzip p6880880_190000_Linux-x86-64.zip -d $ORACLE_HOME
$ORACLE_HOME/OPatch/opatch version

su - oracle
cd $ORACLE_HOME
mv OPatch OPatch_bak
cd /backup/upgrade
# 解压后注意权限
unzip p6880880_190000_Linux-x86-64.zip -d $ORACLE_HOME
$ORACLE_HOME/OPatch/opatch version

5. Conflict verification

5.1. Run OPatch conflict check

GI: Contains multiple packages, check them one by one under the grid user:

$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /backup/upgrade/35037840/35042068

$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /backup/upgrade/35037840/35050331

$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /backup/upgrade/35037840/35050325

$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /backup/upgrade/35037840/35107512

$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /backup/upgrade/35037840/33575402

DB: under the oracle user

$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /backup/upgrade/35037840/35042068
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /backup/upgrade/35037840/35050331

 5.2. Space verification

>>>>使用grid用户检查空间是否满足GIRU部署
$ vi /tmp/patch_list_gihome.txt
/backup/upgrade/35037840/33575402
/backup/upgrade/35037840/35042068
/backup/upgrade/35037840/35050325
/backup/upgrade/35037840/35050331
/backup/upgrade/35037840/35107512
$ORACLE_HOME/OPatch/opatch prereq CheckSystemSpace -phBaseFile /tmp/patch_list_gihome.txt

>>>>使用oracle用户检查系统空间是否满足DBRU的部署
$ vi /tmp/patch_list_dbhome.txt
/backup/upgrade/35037840/35042068
/backup/upgrade/35037840/35050331
$ORACLE_HOME/OPatch/opatch prereq CheckSystemSpace -phBaseFile /tmp/patch_list_dbhome.txt

 5.3. Pre-installation analysis

Use the root user to detect and resolve one-off patch conflicts through the following commands. If the one-off patch conflicts, consider rolling back the patch now, and then consider updating the patch after the PSU/RU is updated.

/u01/app/19.3.0/grid/OPatch/opatchauto apply /backup/upgrade/35037840/ -analyze

/u01/app/19.3.0/grid/OPatch/opatchauto rollback /backup/upgrade/35037840/ -analyze

7. Patch update and verification

7.1.  The GI patches of each node are updated and verified in sequence.

        Notice:

        The upgrade process automatically shuts down and starts the cluster.

        Upgrade the grid of node 1 first, and then upgrade the grid of node 2.

# 使用root用户安装部署GIRU补丁,各个节点依次更新
/u01/app/19.3.0/grid/OPatch/opatchauto apply /backup/upgrade/35037840/ -oh /u01/app/19.3.0/grid
# 部署成功之后进行验证
su - grid
$ORACLE_HOME/OPatch/opatch lspatches
$ORACLE_HOME/OPatch/opatch lsinventory
$ORACLE_HOME/OPatch/opatch lspatches -id 33575402 -verify
$ORACLE_HOME/OPatch/opatch lspatches -id 35042068 -verify
$ORACLE_HOME/OPatch/opatch lspatches -id 35050325 -verify
$ORACLE_HOME/OPatch/opatch lspatches -id 35050331 -verify
$ORACLE_HOME/OPatch/opatch lspatches -id 35107512 -verify

7.2.  The DB patches of each node are updated and verified in sequence.

        Since GI RU includes DB RU, this patch will also be used when upgrading DB in RAC environment.

        Notice:

        The upgrade process automatically shuts down and starts the cluster.

        Upgrade the database of node 1 first, and then upgrade the database of node 2.

# 安装ORACLE RDBMS补丁[使用root用户]
# 使用root用户操作,部署DBRU，各节点依次更新
/u01/app/oracle/product/19.3.0/db_1/OPatch/opatchauto apply /backup/upgrade/35037840/ -oh /u01/app/oracle/product/19.3.0/db_1
# 补丁验证[使用oracle用户]
su - oracle
$ORACLE_HOME/OPatch/opatch lspatches
$ORACLE_HOME/OPatch/opatch lsinventory
$ORACLE_HOME/OPatch/opatch lspatches -id 35042068 -verify

7.3. Update data dictionary

       As mentioned above, updating patches in sequence can reduce downtime, but downtime is still needed, which is the time to run datapatch here. This step is to upgrade the data dictionary, which is for the entire database, so it only needs to be run on one node. The main thing to note is that if it is in cdb mode, you need to alter pluggable database all open, open all pdbs, and then run datapatch.

Non-CDB mode:

% sqlplus /nolog
SQL> Connect / as sysdba
SQL> startup
SQL> quit
% cd $ORACLE_HOME/OPatch
% ./datapatch -verbose

CDB mode:

% sqlplus /nolog
SQL> Connect / as sysdba
SQL> startup
SQL> alter pluggable database all open;
SQL> quit
% cd $ORACLE_HOME/OPatch
% ./datapatch -verbose

 7.4. Handling invalid objects

su - oracle
cd $ORACLE_HOME/rdbms/admin
sqlplus / as sysdba
SQL> @utlrp.sql
 
SQL> select * from product_component_version;
PRODUCT                                  VERSION                                  VERSION_FULL                             STATUS
---------------------------------------- ---------------------------------------- ---------------------------------------- ------------------------------------------------------------
Oracle Database 19c Enterprise Edition   19.0.0.0.0                               19.19.0.0.0                              Production

7.5. Upgrade Oracle Recovery Manager Catalog

If you are using Oracle Recovery Manager, you need to upgrade the catalog. Enter the following command to upgrade. The upgrade catalog command must be entered twice to confirm the upgrade.

$ rman catalog username/password@alias
RMAN> UPGRADE CATALOG;
RMAN> UPGRADE CATALOG;
RMAN> EXIT;

8. View cluster status

crsctl check crs
crsctl check cluster -all
crsctl status res -t