【Environmental introduction】
System environment: Solaris + Oracle 11R2 + OGG + script scheduled task statistics collection
【Background Description】
Group-based security checks require vulnerability scanning of database versions. There are RBDMS and JVM-type vulnerabilities in vulnerability scanning, and minor versions of PSU and OJVM need to be upgraded to fix the vulnerabilities. The following is a brief description of the precautions for upgrading and the solution to the problem.
【Problem handling】
The upgrade is usually performed at night, and the problems that should be paid attention to before upgrading the database (because there are many scripts, here is the script, if you need the script, you can leave a message):
Backup the database software in advance (the aud small files can be excluded, and the backup of small files will lead to a long backup time. It is recommended to back up one day in advance), and data backup (full database backup);
Database parameter backup, parameter comparison after the project is completed (historical reasons lead to different parameters of starting the database pfile file, resulting in exhaustion of host resources or other problems);
Stop timing tasks involving the database (to prevent large transactions from causing problems with the database stop);
Stop the automatic statistics collection of the database or the statistics collection of the manual script (so that the compilation cannot be performed after the upgrade);
Stop other software's operations on the database (OGG, etc., the OGG process is stuck);
The operating system space needs to be greater than 35G (if the space is insufficient, it may trigger a bug:
10131946 GRID PATCHES REQUIRE 22GB OF FREE SPACE
12791141 PATCH 12311357: WITH 35GB FREE SPACE : ERROR: REQUIRED AMOUNT OF SPACE
1088455.1 opatch CheckSystemSpace Fails With Error Code 73 While Applying GI PSU）；
Patch conflicts (DB, some for one of patch conflicts need to be rolled back before upgrading, GI), platform detection.
Before operating the database upgrade, check the database patch conflict and platform detection again, but the patch requires special .so file verification, which is checked after stopping the database instance, which is also a frequent problem of file occupation.
Check information:
grid@XXXDB01:/opt/oracle/app/oracle_base/patch/26636028$ $ORACLE_HOME/OPatch/opatch prereq CheckActiveFilesAndExecutables -phBaseDir ./26635745
Oracle Interim Patch Installer version 11.2.0.3.18
Copyright (c) 2018, Oracle Corporation. All rights reserved.
PREREQ session
Oracle Home : /opt/oracle/app/grid_home
Central Inventory : /opt/oracle/app/oraInventory
from : /opt/oracle/app/grid_home/oraInst.loc
OPatch version : 11.2.0.3.18
OUI version : 11.2.0.4.0
Log file location : /opt/oracle/app/grid_home/cfgtoollogs/opatch/opatch2018-04-27_01-50-50AM_1.log
Invoking prereq "checkactivefilesandexecutables"
......
Prereq "checkActiveFilesAndExecutables" for patch 17478514 failed.
The details are:
Following executables are active :
/opt/oracle/app/grid_home/bin/oracle
/opt/oracle/app/grid_home/lib/libclntsh.so.11.1
Prereq "checkActiveFilesAndExecutables" for patch 18031668 passed.
.......
Prereq "checkActiveFilesAndExecutables" for patch 26392168 passed.
OPatch succeeded.
Omit part of the content, you can see that it is occupied when checking the special file. Check which process is occupied by:
grid@xxxxDB01:/opt/oracle/app/oracle_base/patch/26636028$ fuser /opt/oracle/app/grid_home/lib/libclntsh.so.11.1
/opt/oracle/app/grid_home/lib/libclntsh.so.11.1: 9465m 9447m 9331m 9210m 8889m 8750m 8578m 8498m 8496m 8494m 8417m 8416m 8395m 21708m 8361m 8286m 8164m 14225m 14198m 14186m
Under normal circumstances, the grid user occupying this file will stop these processes when the database is upgraded.
[grid@xxxx01 ~]$ fuser /u01/app/11.2.0/grid/lib/libclntsh.so.11.1
/u01/app/11.2.0/grid/lib/libclntsh.so.11.1: 2400m 2411m 2422m 2432m 2491m 2623m 2837m 2885m 2887m 2984m 2985m 3029m 3046m
[grid@xxxx01 ~]$ ps -ef |egrep '2400|2411|2422|2432|2491|2623|2837|2885|2887|2984|2985|3029|3046'
grid 2400 1 0 10:48 ? 00:00:00 /u01/app/11.2.0/grid/bin/oraagent.bin
grid 2411 1 0 10:48 ? 00:00:00 /u01/app/11.2.0/grid/bin/mdnsd.bin
grid 2422 1 0 10:48 ? 00:00:00 /u01/app/11.2.0/grid/bin/gpnpd.bin
grid 2432 1 0 10:48 ? 00:00:00 /u01/app/11.2.0/grid/bin/gipcd.bin
grid 2491 1 0 10:48 ? 00:00:00 /u01/app/11.2.0/grid/bin/ocssd.bin
grid 2623 1 0 10:49 ? 00:00:00 /u01/app/11.2.0/grid/bin/evmd.bin
grid 2837 2623 0 10:50 ? 00:00:00 /u01/app/11.2.0/grid/bin/evmlogger.bin -o /u01/app/11.2.0/grid/evm/log/evmlogger.info -l /u01/app/11.2.0/grid/evm/log/evmlogger.log
grid 2885 1 0 10:50 ? 00:00:00 /u01/app/11.2.0/grid/bin/scriptagent.bin
grid 2887 1 0 10:50 ? 00:00:00 /u01/app/11.2.0/grid/bin/oraagent.bin
grid 2984 1 0 10:50 ? 00:00:00 /u01/app/11.2.0/grid/opmn/bin/ons -d
grid 2985 2984 0 10:50 ? 00:00:00 /u01/app/11.2.0/grid/opmn/bin/ons -d
grid 3029 1 0 10:50 ? 00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER_SCAN1 -inherit
grid 3046 1 0 10:50 ? 00:00:00 /u01/app/11.2.0/grid/bin/tnslsnr LISTENER -inherit
However, there is a process that is occupied by the root user, and the cluster cannot stop the process:
root@xxxDB01:/opt/oracle/app/oracle_base/patch/26636028# ps -ef |grep /opt/oracle/app/grid_home/jdk/bin/sparcv9/java
root 21708 4111 0 02:37:17 pts/1 0:00 grep /opt/oracle/app/grid_home/jdk/bin/sparcv9/java
【Solution】
About special .so file verification When the file is found to be occupied, use the following method to confirm:
If a file is occupied, kill the occupying process:
opatch error code 73: Prerequisite check "CheckActiveFilesAndExecutables" failed. (文档 ID 1942237.1)
By checking that the OC4J resource is online, special files are occupied by the process. Use crsctl stat res -t to check the OC4J resource status. This resource will lead to 8888 vulnerability. It is recommended to stop this resource. For modifying OC4J, you can check the official document
Security Vulnerability Scan detects Exposed Port on ora.oc4j Resource (文档 ID 1922349.1)
【Summarize】
When upgrading the PSU and OJVM for the database, make sure that the environment has been checked before the upgrade, so that more problems can be encountered during the upgrade process.