--节点1 告警日志
Thu Jun 20 03:02:18 2019
Thread 1 advanced to log sequence 2092 (LGWR switch)
Current log# 11 seq# 2092 mem# 0: +DATA/test/onlinelog/redo11_1.log
Current log# 11 seq# 2092 mem# 1: +DATA/test/onlinelog/redo11_2.log
Thu Jun 20 03:02:18 2019
LNS: Standby redo logfile selected for thread 1 sequence 2092 for destination LOG_ARCHIVE_DEST_2
Thu Jun 20 03:02:20 2019
Archived Log entry 5703 added for thread 1 sequence 2091 ID 0x9b8db1ed dest 1:
Thu Jun 20 03:03:53 2019
skgxpvfynet: mtype: 61 process 316751 failed because of a resource problem in the OS. The OS has most likely run out of buffers (rval: 4)
Errors in file /u01/app/oracle/diag/rdbms/test/test1/trace/test1_ora_316751.trc (incident=560025):
ORA-00603: ORACLE server session terminated by fatal error
ORA-27504: IPC error creating OSD context
ORA-27300: OS system dependent operation:sendmsg failed with status: 105
ORA-27301: OS failure message: No buffer space available
ORA-27302: failure occurred at: sskgxpsnd2
Incident details in: /u01/app/oracle/diag/rdbms/test/test1/incident/incdir_560025/test1_ora_316751_i560025.trc
Thu Jun 20 03:03:54 2019
Dumping diagnostic data in directory=[cdmp_20190620030354], requested by (instance=1, osid=316751), summary=[incident=560025].
opiodr aborting process unknown ospid (316751) as a result of ORA-603
Thu Jun 20 03:04:03 2019
Sweep [inc][560025]: completed
Sweep [inc2][560025]: completed
Thu Jun 20 03:06:15 2019
Thread 1 advanced to log sequence 2093 (LGWR switch)
Current log# 5 seq# 2093 mem# 0: +DATA/test/onlinelog/redo05_1.log
Current log# 5 seq# 2093 mem# 1: +DATA/test/onlinelog/redo05_2.log
Thu Jun 20 03:06:16 2019
---trc 文件--trc文件，初步看应该是服务器 网卡mtu值相关问题
*** 2019-06-20 03:03:53.395
*** CLIENT ID:() 2019-06-20 03:03:53.395
*** SERVICE NAME:() 2019-06-20 03:03:53.395
*** MODULE NAME:() 2019-06-20 03:03:53.395
*** ACTION NAME:() 2019-06-20 03:03:53.395
SKGXP:[7f5ab0ef57c0.0]{0}: SKGXPVFYNET: Socket self-test could not verify successful transmission of 32768 bytes (mtype 61).
SKGXP:[7f5ab0ef57c0.1]{0}: The network is required to support UDP protocol sends of this size. Socket is bound to 169.254.16.188.
SKGXP:[7f5ab0ef57c0.2]{0}: phase 'send', 0 tries, 100 loops, 32354 ms (last)
struct ksxpp * ksxppg_ [0xc11fde0, 0x7f5aadaf5588) = 0x7f5aadaf5580
Dump of memory from 0x00007F5AADAF5580 to 0x00007F5AADAF6AB0
---数据库状态
select INST_ID,INSTANCE_NAME,STARTUP_TIME from gv$instance;
--添加到CRT
set lines 1000\r alter session set nls_date_format='yyyymmdd hh24:mi:ss';\r set time on \r select INST_ID,INSTANCE_NAME,STARTUP_TIME from gv$instance;
09:42:29 SYS@test1(test1)> set lines 1000
09:43:30 SYS@test1(test1)> alter session set nls_date_format='yyyymmdd hh24:mi:ss';
Session altered.
09:43:31 SYS@test1(test1)> set time on
09:43:31 SYS@test1(test1)> select INST_ID,INSTANCE_NAME,STARTUP_TIME from gv$instance;
INST_ID INSTANCE_NAME STARTUP_TIME
---------- ---------------- -----------------
1 test1 20190509 15:30:08
2 test2 20190507 21:37:04
---节点2 日志
Thu Jun 20 03:03:54 2019
Dumping diagnostic data in directory=[cdmp_20190620030354], requested by (instance=1, osid=316751), summary=[incident=560025].
Thu Jun 20 03:10:10 2019
Thread 2 advanced to log sequence 912 (LGWR switch)
Current log# 6 seq# 912 mem# 0: +DATA/test/onlinelog/redo06_1.log
Current log# 6 seq# 912 mem# 1: +DATA/test/onlinelog/redo06_2.log
---参考官方文档
2041723.1
CAUSE
This happens due to less space available for network buffer reservation.
SOLUTION
1. On servers with High Physical Memory, the parameter vm.min_free_kbytes should be set in the order of 0.4% of total Physical Memory. This helps in keeping a larger range of defragmented memory pages available for network buffers reducing the probability of a low-buffer-space conditions.
*** For example, on a server which is having 256GB RAM, the parameter vm.min_free_kbytes should be set to 1073742 ***
On NUMA Enabled Systems, the value of vm.min_free_kbytes should be multiplied by the number of NUMA nodes since the value is to be split across all the nodes.
On NUMA Enabled Systems, the value of vm.min_free_kbytes = n * 0.4% of total Physical Memory. Here 'n' is the number of NUMA nodes.
2. Additionally, the MTU value should be modified as below
#ifconfig lo mtu 16436
To make the change persistent over reboot add the following line in the file /etc/sysconfig/network-scripts/ifcfg-lo :
MTU=16436
Save the file and restart the network service to load the changes
#service network restart
Note : While making the changes in CRS nodes, if network is restarted while CRS is up, it can hung CRS. So cluster services should be stopped prior to the network restart.
---野鸡博客
http://ju.outofmemory.cn/entry/76102