| 问题简述 |
ORA-00240: control file enqueue held for more than 120 seconds ORA-00445: background process "m000" did not start after 120 seconds |
||
| 处理人员 |
xxx |
系统名称 |
xxx报表 |
| 系统版本 |
|
处理地址 |
xxx |
| 数据库版本 |
11.2.0.2 |
数据库模模式 |
RAC |
| 数据库patch |
|
||
| 发生时间 |
Mon Jan 01 17:43:42 2018 |
||
| 问题描述 |
|||
| 1月1日Alert日志报错ORA-00240: control file enqueue held for more than 120 seconds ORA-00445: background process "m000" did not start after 120 seconds |
|||
| 报错信息 |
|||
| Mon Jan 01 17:43:42 2018 |
|||
trace_3605352_i141358.log日志
Description
--------------
*** 2018-01-01 17:43:42.717
*** SESSION ID:(828.1) 2018-01-01 17:43:42.717
*** CLIENT ID:() 2018-01-01 17:43:42.717
*** SERVICE NAME:(SYS$BACKGROUND) 2018-01-01 17:43:42.717
*** MODULE NAME:() 2018-01-01 17:43:42.717
*** ACTION NAME:() 2018-01-01 17:43:42.717
Dump continued from file: /oracle/db/diag/rdbms/ireport/ireport2/trace/ireport2_arc0_3605352.trc
ORA-00240: control file enqueue held for more than 120 seconds
========= Dump for incident 141358 (ORA 240) ========
----- Beginning of Customized Incident Dump(s) -----
-------------------------------------------------------------------------------
CONTROL FILE ENQUEUE HELD FOR TOO LONG
holding mode : S
enqueue holder : 'inst 2, osid 3605352' <<<<<<
enqueue held time : 120 seconds
The current process 'inst 2, osid 3605352' holds the control file enqueue
for more than 120 seconds.
----------------------------------------
SO: 0x7000010d16b6538, type: 4, owner: 0x7000010c13ccbe8, flag: INIT/-/-/0x00 if: 0x3 c: 0x3
proc=0x7000010c13ccbe8, name=session, file=ksu.h LINE:12459 ID:, pg=0
(session) sid: 828 ser: 1 trans: 0x0, creator: 0x7000010c13ccbe8
flags: (0x51) USR/- flags_idl: (0x1) BSY/-/-/-/-/-
flags2: (0x408) -/-
DID: , short-term DID:
txn branch: 0x0
oct: 0, prv: 0, sql: 0x0, psql: 0x0, user: 0/SYS
ksuxds FALSE at location: 0
service name: SYS$BACKGROUND
Current Wait Stack:
1: waiting for 'KSV master wait' <<<<<<<<
=0x0, =0x0, =0x0
wait_id=13855013 seq_num=11861 snap_id=2
wait times: snap=2.493364 sec, exc=2 min 5 sec, total=2 min 5 sec <<<<
wait times: max=infinite, heur=2 min 5 sec
wait counts: calls=41 os=41
in_wait=1 iflags=0x5520
0: waiting for 'Disk file operations I/O'
FileOperation=0x2, fileno=0x0, filetype=0x3
wait_id=13855012 seq_num=11858 snap_id=1
wait times: snap=0.000000 sec, exc=0.000227 sec, total=2 min 5 sec
wait times: max=infinite, heur=2 min 5 sec
wait counts: calls=0 os=0
in_wait=1 iflags=0x15a0
There are 59 sessions blocked by this session.
Dumping one waiter:
inst: 1, sid: 2715, ser: 1
wait event: 'enq: CF - contention'
p1: 'name|mode'=0x43460005
p2: '0'=0x0
p3: 'operation'=0x0
row_wait_obj#: 4294967295, block#: 0, row#: 0, file# 0
min_blocked_time: 104 secs, waiter_cache_ver: 38592
Wait State:
----- Abridged Call Stack Trace -----
ksedsts()+644<-kjzdssdmp()+444<-kjzduptcctx()+272<-kjzdpcrshnfy()+56<-kstdmp()+452<-dbkedDefDump()+10812<-ksedmp()+76<-ksfdmp()+88<
-dbgexPhaseII()+1212<-dbgexExplicitEndInc()+628<-dbgeEndDDEInvocationImpl()+652<-dbgeEndDDEInvocation()+48<-kcc_tac_callback()+2376
<-ksu_dispatch_tac()+1608<-ksvcheckwait()+72<-ksvsend()+3248<-kfncSlaveSendReal()+976<-kfncSlaveSubmitSlv()+920<-kfncFileIdentify()+956<-kfioIdentify()+1860
根据日志分析，该问题是bug 12973375引起的。与bug描述的现象相符
1. call stack is same:
----- Call Stack Trace -----
skdstdst ksedst1 kcc_tac_callback ksu_dispatch_tac ksvcheckwait ksvsend
kfncSlaveSubmitSlv kfncFileIdentify kfioIdentify ksfd_osmopn
ksfdopn1 ksfdopn kcropn kcroio krsk_rlh_get_info krse_arc_source_ini
krse_arc_driver_core krse_arc_driver kcrrwkx kcrrwk
ksbabs krsv_abs ksbrdp opirip opidrv sou2o opimai_real ssthrdmain main
Ct trace call stack:
kcc_tac_callback()+2376<-ksu_dispatch_tac()+1608<-ksvcheckwait()+72<-ksvsend()+3248<-kfncSlaveSendReal()+976
-kfncSlaveSubmitSlv()+920<-kfncFileIdentify()+956<-kfioIdentify()+1860
2.same background process arc hang on same wait event.
Current Wait Stack:
1: waiting for 'KSV master wait' <<<<<<<<
=0x0, =0x0, =0x0
wait_id=13855013 seq_num=11861 snap_id=2
wait times: snap=2.493364 sec, exc=2 min 5 sec, total=2 min 5 sec <<<<
wait times: max=infinite, heur=2 min 5 sec
wait counts: calls=41 os=41
3. same 240 error symptom is shown in trace
CONTROL FILE ENQUEUE HELD FOR TOO LONG
holding mode : S
enqueue holder : 'inst 2, osid 3605352' <<<<<<
enqueue held time : 120 seconds
The current process 'inst 2, osid 3605352' holds the control file enqueue
for more than 120 seconds.
这个问题主要是因为在asm实例上,因为对端进程迟迟不能完成asm操作,导致arc进程出现长时间的
hang住状态,进而长时间持有的控制文件的锁,所以报了 240错误.
建议您打上patch 12830339 ,观察问题是否不再出现.
这个问题目前没有workaround能规避,因为asm实例那边的操作,所以
若是可能因为asm执行某些操作的话,那限制在业务高峰期间类似的ASM操作应该可以避免该问题.
解决方案
Apply patch 12830339 因为 Bug 12973375 is dup on 12830339