RESMGR: cpu quantum wait event processing

Due to a large number of RESMGR:cpu quantum waiting events during the database online process, the resource manager function was closed due to performance problems. The closing process is as follows:
ALTER SYSTEM SET “_resource_manager_always_on”=FALSE SCOPE=SPFILE SID='*';

   execute dbms_scheduler.set_attribute ('SATURDAY_WINDOW','RESOURCE_PLAN','');
   execute dbms_scheduler.set_attribute('SUNDAY_WINDOW','RESOURCE_PLAN','');
   execute dbms_scheduler.set_attribute('MONDAY_WINDOW','RESOURCE_PLAN','');
   execute dbms_scheduler. set_attribute('TUESDAY_WINDOW','RESOURCE_PLAN','');
   execute dbms_scheduler.set_attribute('WEDNESDAY_WINDOW','RESOURCE_PLAN','');
   execute dbms_scheduler.set_attribute('THURSDAY_WINDOW','RESOURCE_PLAN','' );
   execute dbms_scheduler.set_attribute('FRIDAY_WINDOW','RESOURCE_PLAN','');

#4 Restart the database to take effect

After the restart, the resource manager is closed, and the database no longer has the RESMGR:cpu quantum wait event.
But the database automatic task plan scheduling starts to report an error, and the background error is as follows:
ORA-12012: error on auto execute of job "SYS"."ORA$AT_SQ_SQL_SW_186"
ORA-29373: resource manager is not on

view check error:
SQL> select client_name, window_name,job_name,job_status,job_start_time from dba_autotask_job_history ;

CLIENT_NAME WINDOW_NAME JOB_NAME JOB_STATUS JOB_START_TIME
-------------------- --------------- ----- ------------------------------ --------------- --------------- ----------------------------------- -----
auto optimizer stats WEDNESDAY_WINDOW ORA$AT_OS_OPT_SY_102 FAILED 09-APR-14 10.00.01.582006 PM PRC
collection

auto optimizer stats FRIDAY_WINDOW ORA$AT_OS_OPT_SY_105 FAILED 11-APR-14 10.00.06.999466 PM PRC
collection

auto optimizer stats TUESDAY_WINDOW ORA$AT_OS_OPT_SY_99 FAILED 08-APR-14 10.00.07.885416 PM PRC
collection Open 29373 errorstack to report an error, and collect the detailed trace log in the error report: the background alert reports the following error: Sun May 04 06:00:04 2014 Dumping diagnostic data in directory=[cdmp_20140504060004], requested by (instance=1, osid=55167 ( J001)), summary=[abnormal process termination]. Errors in file /oracle/database/diag/rdbms/nfdb/nfdb1/trace/nfdb1_j001_55167.trc: ORA-29373: resource manager is not on









Errors in file /oracle/database/diag/rdbms/nfdb/nfdb1/trace/nfdb1_j001_55167.trc:
ORA-12012: error on auto execute of job "SYS"."ORA$AT_OS_OPT_SY_271"
ORA-29373: resource manager is not on
Dumping diagnostic data in directory=[cdmp_20140504060009], requested by (instance=1, osid=55167 (J001)), summary=[abnormal process termination].


The detailed trace report is as follows:
dbkedDefDump(): Starting a non-incident diagnostic dump (flags=0x0, level=3, mask=0x0)
----- Error Stack Dump -----
ORA-29373: resource manager is not on
----- SQL Statement (None) ---- -
Current SQL information unavailable - no cursor.

From the trace file, we can only confirm that the JOB cannot be scheduled because of the resource manager.

However , during the inspection, we found that the process DBRM that manages the resource manager still exists in the database.
#ps –ef | grep dbrm
oracle 34920 1 0 Apr18 ? 00:00:59 ora_dbrm_nfdb1


This gives us a certain direction of suspicion to a certain extent, there may be a resource that is not completely closed, and from the detailed trace file from the call stack From the trace, the current process has no function calls about the resource manager, but has been posting messages to another process.
So, we suspect that the error is most likely caused by DBRM not shutting down properly.

The above analysis results are now judged, and the acquisition may be based on

four fault simulations. The

fault simulation is divided into the following parts:
Step operation results
Step1 is the same as the fault operation, set the implicit parameter of "_resource_manager_always_on, close the resource manager windows call plan background error
Step2 delete Implicit parameters, only set the resource manager windows call plan to close the background without error
Step3 Add 2 implicit parameters, close the resource manager windows call plan background without error The

detailed test process is as follows:
For the test, we set the database time to 22 o'clock and let it Automatic scheduling of JOB plan
4.1 Simulate the failure environment of the management library
Shut down the database, adjust the time, set implicit parameters, close the windows plan
Shut down the database:
SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.

Adjusted time:
[root@rhel6 ~]# date -s 2014/05/14
[root@rhel6 ~]# date -s 21:57:00
[root@rhel6 ~] # clock -w
[root@rhel6 ~]# date
Wed May 14 21:57:35 CST 2014

Add implicit parameters, start to open:
SQL> ALTER SYSTEM SET "_resource_manager_always_on"=FALSE SCOPE=SPFILE;
SQL> startup force( Test environment, directly force start, do not do this in production environment)

Set resource manager plan:
execute dbms_scheduler.set_attribute('SATURDAY_WINDOW','RESOURCE_PLAN','');
execute dbms_scheduler.set_attribute('SUNDAY_WINDOW','RESOURCE_PLAN','' );
execute dbms_scheduler.set_attribute('MONDAY_WINDOW',' RESOURCE_PLAN','');
execute dbms_scheduler.set_attribute('TUESDAY_WINDOW','RESOURCE_PLAN','');
execute dbms_scheduler.set_attribute('WEDNESDAY_WINDOW','RESOURCE_PLAN','');
execute dbms_scheduler.set_attribute('THURSDAY_WINDOW','RESOURCE_PLAN','');
execute dbms_scheduler.set_attribute('FRIDAY_WINDOW','RESOURCE_PLAN','');

我们观察22点的alert信息，确实开始报错：
Wed May 14 22:00:03 2014
Errors in file /oracle/ora11g/base/diag/rdbms/ora11g/ora11g/trace/ora11g_j003_4452.trc:
ORA-12012: error on auto execute of job "SYS"."ORA$AT_OS_OPT_SY_27"
ORA-29373: resource manager is not on
Wed May 14 22:00:03 2014
Errors in file /oracle/ora11g/base/diag/rdbms/ora11g/ora11g/trace/ora11g_j004_4454.trc:
ORA-12012: error on auto execute of job "SYS"."ORA$AT_SA_SPC_SY_28"
ORA-29373: resource manager is not on
Wed May 14 22:00:03 2014
Errors in file /oracle/ora11g/base/diag/rdbms /ora11g/ora11g/trace/ora11g_j005_4456.trc:
ORA-12012: error on auto execute of job "SYS"."ORA$AT_SQ_SQL_SW_29"
ORA-29373: resource manager is not on
Wed May 14 22:00:04 2014
XDB installed .
XDB initialized.

Check that the DBRM process already exists:
[root@rhel6 ~]# ps -ef | grep dbrm
ora11g 4346 1 0 21:57 ? 00:00:00 ora_dbrm_ora11g


Check the background JOB execution record view:
SQL> select CLIENT_NAME,WINDOW_NAME,JOB_NAME,JOB_STATUS,JOB_START_TIME from DBA_AUTOTASK_JOB_HISTORY where CLIENT_NAME='auto optimizer stats collection' order by JOB_START_TIME desc;

CLIENT_NAME WINDOW_NAME JOB_NAME JOB_STATUS JOB_START_TIME
----------------- ------------------ ---------------------------------------- -- ---------------------------- ---------------------- -------- ----------------------------------------
auto optimizer stats collection WEDNESDAY_WINDOW ORA$AT_OS_OPT_SY_27 FAILED 14-MAY-14 10.00.03.233570 PM The PRC

simulation process is consistent with the production management library, and the results are also consistent. The DBRM process already exists.


4.2 Simulate the removal of implicit parameters
Close database, adjust the time, remove the implicit parameters, close the windows plan
Close the database:
SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.

Adjust time:
[root@rhel6 ~]# date -s 2014/05/14
[root@rhel6 ~]# date -s 21:57:00
[ root@rhel6 ~]# date -s 2014/05/14 @rhel6 ~]# clock -w
[root@rhel6 ~]# date
Wed May 14 21:57:35 CST 2014

Remove implicit parameters, start to open:
remove implicit parameters using create pfile from spfile;
delete spfile, edit pfile file, delete the implicit parameters, and start the database with pfile

Set the resource manager plan: (since it has been set before, no need to set it again)

We observed the alert information at 22 o'clock and found that there was no error
Tue May 13 22:00:03 2014
Begin automatic SQL Tuning Advisor run for special tuning task "SYS_AUTO_SQL_TUNING_TASK"
End automatic SQL Tuning Advisor run for special tuning task "SYS_AUTO_SQL_TUNING_TASK"
Tue May 13 22:00:05 2014
XDB installed.
XDB initialized.


Check DBRM process:
[root@rhel6 ~]# ps -ef | grep dbrm
ora11g 3844 1 0 21:55 ? 00:00:00 ora_dbrm_ora11g

Description: At this time Because the resource manager only closes the resource manager plan plan, it does not actually close the resource manager, so the process still exists.

Check background JOB execution view information:
SQL> select CLIENT_NAME,WINDOW_NAME,JOB_NAME,JOB_STATUS,JOB_START_TIME from DBA_AUTOTASK_JOB_HISTORY where CLIENT_NAME='auto optimizer stats collection' order by JOB_START_TIME desc;

CLIENT_NAME WINDOW_NAME JOB_NAME JOB_STATUS JOB_START_TIME
---------------------------------------- ---------- -------------------- ------------------------- ----- --------------- ----------------------------------- -----
auto optimizer stats collection TUESDAY_WINDOW ORA$AT_OS_OPT_SY_24 SUCCEEDED 13-MAY-14 10.00.02.102741 PM PRC

shows that when the implicit parameters are removed, JOB can be executed normally, and no error is reported in the background.

4.3 Simulate the existence of implicit parameters and resource manager plan
Close database, adjust the time, add implicit parameters, close the windows plan
Close the database:
SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
Adjust time:
[root @rhel6 ~]# date -s 2014/05/15
[root@rhel6 ~]# date -s 21:57:00
[root@rhel6 ~]# clock -w
[root@rhel6 ~]# date
Thu May 15 21:57:35 CST 2014
Add implicit parameters, start to open:
SQL> alter system set "_resource_manager_always_off"=true scope=spfile;
SQL> ALTER SYSTEM SET "_resource_manager_always_on"=FALSE SCOPE=SPFILE;
SQL> startup force (test environment, direct force start, do not do this in production environment)
set resource manager plan: (because it has been set before, no need to set again)

We observed the alert information at 22 o'clock and found that no error was reported
Thu May 15 22:00:03 2014
Begin automatic SQL Tuning Advisor run for special tuning task "SYS_AUTO_SQL_TUNING_TASK"
Thu May 15 22:00:05 2014
XDB installed.
XDB initialized.
End automatic SQL Tuning Advisor run for special tuning task "SYS_AUTO_SQL_TUNING_TASK"

Check DBRM process:
[root@rhel6 ~]# ps -ef | grep dbrm
root 4650 3647 0 21:58 pts/2 00:00:00 grep dbrm

description: DBRM process disappeared at this time

Check background JOB execution view information:
SQL> select CLIENT_NAME,WINDOW_NAME,JOB_NAME,JOB_STATUS,JOB_START_TIME from DBA_AUTOTASK_JOB_HISTORY where CLIENT_NAME='auto optimizer stats collection' order by JOB_START_TIME desc;

CLIENT_NAME WINDOW_NAME JOB_NAME JOB_STATUS JOB_START_TIME
----------------------------------- - ----------------------------- --------------------- --------- ------------------------------ ------------- -----------------------------
auto optimizer stats collection THURSDAY_WINDOW ORA$AT_OS_OPT_SY_30 SUCCEEDED 15-MAY-14 10.00.02.115232 PM PRC

This shows that when both implicit parameters are added and the resource manager is completely shielded, the JOB can be executed normally, and no error is reported in the background.


Five summary description The
above test results prove that the reason for the failure of the JOB execution in the background error should be that the DBRM process is still active, and the DBRM process is the management RESOURCE manager. When removing "_resource_manager_always_off"=true and "_resource_manager_always_on"=FALSE
or adding both parameters, both This error can be avoided, and statistics can be automatically collected and executed automatically.
Because managing the database is relatively important, and there has been a serious RESMGR:cpu quantum wait event when going online, it is recommended not to remove the implicit parameter, but to add "_resource_manager_always_off"=true implicitly. With parameters, restart the database

Of course , you can confirm on the other two sets of libraries first, and then operate on the management library