Remember once Oracle session number caused by excessive number of processes than processes ultimately lead to problems limit the client can not connect

RAC suddenly alert, the client will be reported when trying to connect: ORA-12520: TNS: listener could not find available handler for requested type of server error.

Now finishing the settlement process for quick processing in the subsequent encounter this problem.

1. Log in to the server, using sqlplus / as sysdba can find access to the database.

2. Check the alert log, the log has found error message: ORA-00020: maximum number of processes (2000) exceeded, thereby determining the number of process is super.

Alert log path can be found by sql:

select * from gv$diag_info where name like '%Alert%'

3. Determine the Oracle user's problem,

select t1.inst_id,t1.username,count(*) from gv$session t1
 join gv$process t2 on t2.addr=t1.paddr and t2.inst_id=t1.inst_id
 group by t1.inst_id,t1.username order by 3 desc

Can be found is too much xxxx number of user connections caused.

4. prevent users from re-establish a new connection.

Depending on the degree of control of the client to choose a flexible approach. If you have absolute control over the client program, the problem of the program can be stopped, under normal circumstances, the problem will be solved.

I chose the way of the user lock, do the following sql:

alter user xxxx account lock;

5. At this point we see the gv $ session, found that the connection has been established still.

select * from gv$session where username='XXXX'

6. clean up these connections.

At this time, do not use alter system kill session, because after my follow-up tests found that if the problem does not quit the client program, a network connecting the client and Oracle server is established has remained at this time even if we kill session, process still will not be released, it will lead to gv $ session view and gv $ process view of the lost relationship, which led to us finding out the user session corresponding to those processes.

Directly kill system processes the corresponding session in the operating system level.

XXXX can be found using the following sql account of the session corresponding system process ID:

select t1.inst_id,t1.sid,t1.serial#,t1.username,t2.spid 系统进程id from gv$session t1
 join gv$process t2 on t2.addr=t1.paddr and t2.inst_id=t1.inst_id
 where t1.username='XXXX'

7. occupied finish above operation process to come down. Then we consider the program to communicate with the client stopped the problem and then to lock out the account were restored.

All the way to kill a given session corresponding operating system commands, this method is too violent, do not use as much as possible.

ps -ef | grep LOCAL = NO | grep -v grep | awk '{print $ 2}' | xargs kill -9

 

 

 

After the issue is resolved, I use python to do a little test in the library reproduces the problem, as follows:

#!/bin/python
# -*- coding: utf-8 -*-
import cx_Oracle as co
conn=co.SessionPool('XXXX','XXXXXXXXX','x.x.x.x',min=1,max=5000)

a=[]
for i in range(5000):
    print(i)
    try:
        ccc=conn.acquire()
        a.append(ccc)
    except Exception as e:
        print(e)
        time.sleep(20)

 

the above.