VCS引起的oracle数据库异常重启一例

Oracle 2020-06-23

1. 环境描述

操作系统版本：SUSELinuxEnterprise Server 10 sp2 (x86_64)

数据库版本：Oracle11.1.0.7.16

VCS版本：5.1

blog地址：http://blog.csdn.net/hw_libo/article/details/41171561

2. 问题现象及分析

凌晨3:46左右，手机短信收到数据库异常告警。

（1）查看数据库alert日志

[html]view plaincopy 
   
print?
 SunNov1603:46:512014
 StoppingbackgroundprocessSMCO
 StoppingbackgroundprocessFBDA
 Shuttingdowninstance:furtherlogonsdisabled
 SunNov1603:46:542014
 kkjcre1p:unabletospawnjobqslaveprocess,slot0,error1089
 StoppingbackgroundprocessCJQ0
 StoppingbackgroundprocessQMNC
 StoppingbackgroundprocessMMNL
 StoppingbackgroundprocessMMON
 SunNov1603:47:032014
 Shuttingdowninstance(immediate)
 Licensehighwatermark=366
 SunNov1603:51:522014
 USER(ospid:8654):terminatingtheinstance
 SunNov1603:51:522014
 ORA-1092:opiodrabortingprocessunknownospid(28656_47367034883680)
 LGWRwaitingforinstancetermination

并未发现较为明显的异常原因。

同时也查看了那个时间段的报告，发现数据库压力不大，也没有什么异常的sql存在。

（2）检查OS资源使用

系统资源使用正常，IO也不高。

（3）查看messages日志

copy 
    
   
Nov1603:42:46boscodbAgentFramework[10894]:VCSERRORV-16-2-13027Thread(4149214112)Resource(ora_vg)-monitorproceduredidnotcompletewithintheexpectedtime. Nov1603:42:46boscodbHad[10258]:VCSERRORV-16-2-13027(boscodb)Resource(ora_vg)-monitorproceduredidnotcompletewithintheexpectedtime.
 Nov1603:43:01boscodbsu:(toroot)rootonnone
 ……
 Nov1603:45:39boscodbsu:(tooracle)rootonnone
 Nov1603:45:47boscodbAgentFramework[10894]:VCSERRORV-16-2-13210Thread(4154813344)Agentiscallingcleanforresource(ora_vg)because4successiveinvocationsofthemonitorproceduredidnotcompletewithintheexpectedtime.
 Nov1603:45:47boscodbHad[10258]:VCSERRORV-16-2-13210(boscodb)Agentiscallingcleanforresource(ora_vg)because4successiveinvocationsofthemonitorproceduredidnotcompletewithintheexpectedtime.
 Nov1603:46:48boscodbAgentFramework[10894]:VCSERRORV-16-2-13006Thread(4153760672)Resource(ora_vg):cleanproceduredidnotcompletewithintheexpectedtime.
 Nov1603:46:48boscodbHad[10258]:VCSERRORV-16-2-13006(boscodb)Resource(ora_vg):cleanproceduredidnotcompletewithintheexpectedtime.
 Nov1603:46:49boscodbAgentFramework[10899]:VCSERRORV-16-2-13067Thread(4155194272)Agentiscallingcleanforresource(ora_archive)becausetheresourcebecameOFFLINEunexpectedly,onitsown.
 Nov1603:46:50boscodbHad[10258]:VCSERRORV-16-2-13067(boscodb)Agentiscallingcleanforresource(ora_archive)becausetheresourcebecameOFFLINEunexpectedly,onitsown.
 Nov1603:46:51boscodbAgentFramework[10899]:VCSERRORV-16-2-13068Thread(4155194272)Resource(ora_archive)-cleancompletedsuccessfully.

原来是VCS出现了错误，监控vg超时了，这些错误信息在VCS的日志中也有记录。

3. 解决办法

检查了监控vg的moniter脚本，发现也没有异常，而且脚本在其他库，压力都比较大的都不曾出问题。

当前的配置为：

copy 
    
   
Applicationora_vg( StartProgram="/opt/BOSCO_ora/bin/vg_start.sh"
 StopProgram="/opt/BOSCO_ora/bin/vg_stop.sh"
 CleanProgram="/opt/BOSCO_ora/bin/vg_clean.sh"
 MonitorProgram="/opt/BOSCO_ora/bin/vg_monitor.sh"
 MonitorInterval=30
 MonitorTimeout=30
 )