First, the failure phenomenon
15: 30-16: 00 testing found that a large number of enq: HW - contention and enq: TX - row lock contention wait for the event, and the two events more and more, eventually leading to a short service downtime.
Back testing found uses table space is being used to automatically expand, thus increasing users table space, problem solving
Second, failure analysis
1, see 15: 30-16: 00 time snap_id, used to locate all this period of time the session operation
select * from dba_hist_snapshot where begin_interval_time>to_date('2019-12-03','yyyy-mm-dd')
2, waiting to see the main event this time
select to_char(sample_time, 'YYYY-MM-DD HH24:MI'),event, count(*) from dba_hist_active_sess_history where snap_id in (116604, 116605) and sample_time>=to_date('2019-12-03 15:10','yyyy-mm-dd hh24:mi') group by to_char(sample_time, 'YYYY-MM-DD HH24:MI'),event order by 1,3;
From the above analysis it can be seen as wait event enq: HW - contention, enq: TX - row lock contention
3. The above statement is waiting to see which events generated by the execution (526awsta39934)
select t.instance_number, t.sample_time, t.session_id, t.session_serial#, t.sql_id, t.event, t.blocking_session, t.blocking_inst_id, t.program, t.machine from dba_hist_active_sess_history t where snap_id in (116604, 116605) and to_char(sample_time,'yyyy-mm-dd hh24:mi')='2019-12-03 15:39' and event in ('enq: TX - row lock contention', 'enq: HW - contention') order by 2, 1;
4, according to the above sql_id as '526awsta39934' caused 'enq: HW - contention' high-water mark contention event to view the statement
INSERT INTO TRANSACTION_LOG VALUES (:B1 ,:B2 ,:B3 ,:B4 ,:B5 ,:B6 ,:B7,:B8 ,:B9 ,:B10 ,:B11 ,:B12 ,:B13 ,:B14 ,:B15 )
5, 3, can be isolated according to step 'enq: TX - row lock contention' line lock blocking session id
Obstruction above the session id can look for according to exactly the sql_id '526awsta39934' blocked
select t.instance_number, t.sample_time, t.session_id, t.session_serial#, t.sql_id, t.event, t.blocking_session, t.blocking_inst_id, t.program, t.machine from dba_hist_active_sess_history t where snap_id in (116604, 116605) and /*sample_time >= to_date('2019-12-03 15:10', 'yyyy-mm-dd hh24:mi') and */to_char(sample_time,'yyyy-mm-dd hh24:mi')='2019-12-03 15:39' and t.session_id in (2531,3595,2128,3208,3555,5598,168) and t.instance_number=2 -- and event in ('enq: TX - row lock contention', 'enq: HW - contention') order by 2, 1;
6, summed up
From the above analysis can be high when a large number of concurrent statement '526awsta39934', inserting data into the database will be available to view the data block blocks in high-water mark, just at this time is automatically extended tablespace users, while the other session data are inserted or modify the data, need to be used at the high-water mark blocks. Eventually caused a row lock.
ps: the reference high-water mark on oracle
https://www.cnblogs.com/linjiqin/archive/2012/01/15/2323030.html