bug 7207932 MRP0 Background Media Recovery terminated with error 600 ora-00600 kgeade_is_0
May192014
今天在给客户的一套10.2.0.4.0的RAC搭建DG的时候，又遇到了BUG，主库使用ASM文件系统，备库单实例环境，直接使用文件系统，在启动MRP进程应用归档的时候，MRP进程遇到600错误而终止。
通过ps命令查看不到mrp进程：
[oracle@b2-jjywfkdb oracle]$ ps -ef | grep mrp oracle 20297 1 0 14:49 ? 00:00:00 ora_mrp0_head2bak oracle 20661 20003 0 15:15 pts/3 00:00:00 grep mrp
告警日志出现600错误：
Fri May 9 14:45:39 2014 alter database recover managed standby database disconnect from session Fri May 9 14:45:39 2014 Attempt to start background Managed Standby Recovery process (head2bak) MRP0 started with pid=16, OS id=20115 Fri May 9 14:45:39 2014 MRP0: Background Managed Standby Recovery process started (head2bak) Managed Standby Recovery not using Real Time Apply parallel recovery started with 15 processes Fri May 9 14:45:46 2014 Waiting for all non-current ORLs to be archived... Fri May 9 14:45:46 2014 Errors in file /oracle/admin/head2bak/bdump/head2bak_mrp0_20115.trc: ORA-00313: open failed for members of log group 11 of thread 1 Fri May 9 14:45:46 2014 Errors in file /oracle/admin/head2bak/bdump/head2bak_mrp0_20115.trc: ORA-00313: open failed for members of log group 11 of thread 1 Clearing online redo logfile 11 +DATA2014 Clearing online log 11 of thread 1 sequence number 1674 Fri May 9 14:45:46 2014 Errors in file /oracle/admin/head2bak/bdump/head2bak_mrp0_20115.trc: ORA-00313: open failed for members of log group 11 of thread 1 Fri May 9 14:45:46 2014 Errors in file /oracle/admin/head2bak/bdump/head2bak_mrp0_20115.trc: ORA-00600: internal error code, arguments: [kgeade_is_0], [], [], [], [], [], [], [] Fri May 9 14:45:46 2014 Completed: alter database recover managed standby database disconnect from session Fri May 9 14:45:48 2014 Errors in file /oracle/admin/head2bak/bdump/head2bak_mrp0_20115.trc: ORA-00600: internal error code, arguments: [kgeade_is_0], [], [], [], [], [], [], [] Fri May 9 14:45:48 2014 MRP0: Background Media Recovery terminated with error 600 Fri May 9 14:45:48 2014 Errors in file /oracle/admin/head2bak/bdump/head2bak_mrp0_20115.trc: ORA-00600: internal error code, arguments: [kgeade_is_0], [], [], [], [], [], [], [] Fri May 9 14:45:48 2014 Errors in file /oracle/admin/head2bak/bdump/head2bak_mrp0_20115.trc: ORA-00600: internal error code, arguments: [kgeade_is_0], [], [], [], [], [], [], [] Fri May 9 14:45:48 2014 Errors in file /oracle/admin/head2bak/bdump/head2bak_mrp0_20115.trc: ORA-00600: internal error code, arguments: [kgeade_is_0], [], [], [], [], [], [], []
在应用归档时，发现REDO LOG不存在，ORACLE会CLRAE出新的REDO LOG，而CLEAR时发现路径指定的是ASM，而备库没有使用ASM，就CLEAR失败了。可是数据库已经设置了LOG FILE CONVERT参数，正常会自动在CONVERT指定的目录创建日志文件，为什么没有CLEAR呢？
查看v$LOGFILE视图，发现以下情况。
GROUP# MEMBER
---------- ----------------------------------------------------------------------
1 /backups_20140506/head2bak/redo01.log
2 /backups_20140506/head2bak/redo02.log
3 /backups_20140506/head2bak/redo03.log
11 +DATA
11 +DATA2014
11 /backups_20140506/head2bak/red011.log
15 /backups_20140506//headdb2/onlinelog/group_15.273.846004339
15 /backups_20140506/2014/headdb2/onlinelog/group_15.264.846004319
15 /backups_20140506/head2bak/red015.log
20 /backups_20140506/head2bak/redo20.log
21 /backups_20140506/head2bak/redo21.log
22 /backups_20140506/head2bak/redo22 .log
23 /backups_20140506/head2bak/redo23.log
以上，123组是手动加的ONLINE REDO LOG，11和15组是指定到ASM的ONLINE REDO LOG，其他是手动创建的STANDBY REDO LOG。
第15组是ACTIVE状态的ONLINE REDO LOG，第11组是CURRENT ONLINE REDO LOG，可见ACTIVE状态的日志已经在CONVERT目录下成功CLEAR出来了，CURRENT(第11组)没有CONVERT，于是抛出600错误，经过查看MOS，确定遇到了ORACLE的BUG(BUG:7207932 – ORA-600 [KGEADE_IS_0] WHEN RENAMING A FILE FROM ASM TO FS？ A possible workaround is to recreate the controlfile and specifying new filenames in the controlfile. If this does not work, then apply the patch for Bug:7207932.)。这个BUG已经在7207932补丁修复。那就安装7207932补丁尝试解决。
[oracle@b2-jjywfkdb ~]$ ls p7207932_10204_Linux-x86-64.zip script [oracle@b2-jjywfkdb ~]$ unzip p7207932_10204_Linux-x86-64.zip [oracle@b2-jjywfkdb ~]$ cd 7207932/ [oracle@b2-jjywfkdb 7207932]$ /oracle/product/10.2.0/db_1/OPatch/opatch apply Invoking OPatch 10.2.0.4.2 Oracle Interim Patch Installer version 10.2.0.4.2 Copyright (c) 2007, Oracle Corporation. All rights reserved. Oracle Home : /oracle/product/10.2.0/db_1 Central Inventory : /oracle/oraInventory from : /etc/oraInst.loc OPatch version : 10.2.0.4.2 OUI version : 10.2.0.4.0 OUI location : /oracle/product/10.2.0/db_1/oui Log file location : /oracle/product/10.2.0/db_1/cfgtoollogs/opatch/opatch2014-05-09_14-46-39PM.log ApplySession applying interim patch '7207932' to OH '/oracle/product/10.2.0/db_1' Running prerequisite checks... OPatch detected non-cluster Oracle Home from the inventory and will patch the local system only. Please shutdown Oracle instances running out of this ORACLE_HOME on the local system. (Oracle Home = '/oracle/product/10.2.0/db_1') Is the local system ready for patching? [y|n] y User Responded with: Y Backing up files and inventory (not for auto-rollback) for the Oracle Home Backing up files affected by the patch '7207932' for restore. This might take a while... Backing up files affected by the patch '7207932' for rollback. This might take a while... Patching component oracle.rdbms, 10.2.0.4.0... Updating archive file "/oracle/product/10.2.0/db_1/lib/libserver10.a" with "lib/libserver10.a/kfn.o" Updating archive file "/oracle/product/10.2.0/db_1/lib/libserver10.a" with "lib/libserver10.a/kfnc.o" Running make for target ioracle ApplySession adding interim patch '7207932' to inventory Verifying the update... Inventory check OK: Patch ID 7207932 is registered in Oracle Home inventory with proper meta-data. Files check OK: Files from Patch ID 7207932 are present in Oracle Home. The local system has been patched and can be restarted. OPatch succeeded.
安装完补丁后，验证补丁是否成功安装。
[oracle@b2-jjywfkdb 7207932]$ /oracle/product/10.2.0/db_1/OPatch/opatch lsinventory
Invoking OPatch 10.2.0.4.2
Oracle Interim Patch Installer version 10.2.0.4.2
Copyright (c) 2007, Oracle Corporation. All rights reserved.
Oracle Home : /oracle/product/10.2.0/db_1
Central Inventory : /oracle/oraInventory
from : /etc/oraInst.loc
OPatch version : 10.2.0.4.2
OUI version : 10.2.0.4.0
OUI location : /oracle/product/10.2.0/db_1/oui
Log file location : /oracle/product/10.2.0/db_1/cfgtoollogs/opatch/opatch2014-05-09_14-48-19PM.log
Lsinventory Output file location : /oracle/product/10.2.0/db_1/cfgtoollogs/opatch/lsinv/lsinventory2014-05-09_14-48-19PM.txt
--------------------------------------------------------------------------------
Installed Top-level Products (2):
Oracle Database 10g 10.2.0.1.0
Oracle Database 10g Release 2 Patch Set 3 10.2.0.4.0
There are 2 products installed in this Oracle Home.
Interim patches (1) :
Patch 7207932 : applied on Fri May 09 14:47:15 CST 2014
Created on 14 Oct 2008, 06:53:50 hrs PST8PDT
Bugs fixed:
7207932
--------------------------------------------------------------------------------
OPatch succeeded.
确认已经成功安装补丁后，重新启动mrp进程，mrp进程可正常启动。
Fri May 9 14:49:54 2014 alter database recover managed standby database disconnect from session Fri May 9 14:49:54 2014 Attempt to start background Managed Standby Recovery process (head2bak) MRP0 started with pid=16, OS id=20297 Fri May 9 14:49:54 2014 MRP0: Background Managed Standby Recovery process started (head2bak) Managed Standby Recovery not using Real Time Apply parallel recovery started with 15 processes Fri May 9 14:50:01 2014 Waiting for all non-current ORLs to be archived... Fri May 9 14:50:01 2014 Errors in file /oracle/admin/head2bak/bdump/head2bak_mrp0_20297.trc: ORA-00313: open failed for members of log group 11 of thread 1 Fri May 9 14:50:01 2014 Errors in file /oracle/admin/head2bak/bdump/head2bak_mrp0_20297.trc: ORA-00313: open failed for members of log group 11 of thread 1 Clearing online redo logfile 11 +DATA2014 Clearing online log 11 of thread 1 sequence number 0 Fri May 9 14:50:01 2014 Errors in file /oracle/admin/head2bak/bdump/head2bak_mrp0_20297.trc: ORA-00313: open failed for members of log group 11 of thread 1 Fri May 9 14:50:01 2014 Errors in file /oracle/admin/head2bak/bdump/head2bak_mrp0_20297.trc: ORA-00349: failure obtaining block size for '+DATA2014' Clearing online redo logfile 11 complete Media Recovery Waiting for thread 1 sequence 1088 Fetching gap sequence in thread 1, gap sequence 1088-1187 Fri May 9 14:50:01 2014 Completed: alter database recover managed standby database disconnect from session
由于GAP的归档在主库已经删掉，利用RMAN从备份中恢复这些归档，看看MRP进程是否可以应用。
run{
allocate channel d1 type disk;
allocate channel d2 type disk;
allocate channel d3 type disk;
allocate channel d4 type disk;
allocate channel d5 type disk;
allocate channel d6 type disk;
set archivelog destination to '/oradata/head2bak/arch';
restore archivelog from logseq 1088 until logseq 1187;
release channel d1;
release channel d2;
release channel d3;
release channel d4;
release channel d5;
release channel d6;
}
Fri May 9 15:01:35 2014
Archivelog restore complete. Elapsed time: 0:03:27
Archivelog restore complete. Elapsed time: 0:03:28
Archivelog restore complete. Elapsed time: 0:03:29
Archivelog restore complete. Elapsed time: 0:03:29
Archivelog restore complete. Elapsed time: 0:03:30
Archivelog restore complete. Elapsed time: 0:03:31
Fri May 9 15:04:26 2014
Archivelog restore complete. Elapsed time: 0:02:51
Fri May 9 15:07:07 2014
Archivelog restore complete. Elapsed time: 0:02:41
Fri May 9 15:09:25 2014
Archivelog restore complete. Elapsed time: 0:02:16
Fri May 9 15:12:14 2014
Archivelog restore complete. Elapsed time: 0:02:51
Archivelog restore complete. Elapsed time: 0:02:51
Archivelog restore complete. Elapsed time: 0:02:52
Archivelog restore complete. Elapsed time: 0:02:53
Fri May 9 15:15:32 2014
Archivelog restore complete. Elapsed time: 0:03:15
Archivelog restore complete. Elapsed time: 0:03:18
Archivelog restore complete. Elapsed time: 0:03:19
Fri May 9 15:16:02 2014
Media Recovery Log /oradata/head2bak/arch/1_1088_845998292.dbf
Fri May 9 15:16:51 2014
Archivelog restore complete. Elapsed time: 0:01:19
Archivelog restore complete. Elapsed time: 0:01:20
Fri May 9 15:20:06 2014
Media Recovery Log /oradata/head2bak/arch/1_1089_845998292.dbf
Fri May 9 15:20:54 2014
Archivelog restore complete. Elapsed time: 0:02:49
Fri May 9 15:23:32 2014
Archivelog restore complete. Elapsed time: 0:02:31
Archivelog restore complete. Elapsed time: 0:02:38
Fri May 9 15:25:38 2014
Media Recovery Log /oradata/head2bak/arch/1_1090_845998292.dbf
通过观察，应用归档已经正常。只是很郁闷，最近做的项目怎么都能遇到BUG。
———————————————-end———————————————–
【上一篇】bug 4679769 Failed to upgrade Oracle Cluster Registry configuration错误
【下一篇】sculkget failed to lock oracle10gdbdbslkinstB1ACDB2 exclusive lock held by PID
【下一篇】sculkget failed to lock oracle10gdbdbslkinstB1ACDB2 exclusive lock held by PID
bug 7207932 MRP0 Background Media Recovery terminated with error 600 ora-00600 kgeade_is_0：等您坐沙发呢！
发表评论
