13518219792
1. [grid@RAC01 ~]$ crsctl stop database -d RACwh
数据库关闭ok后此时我们进行关闭crs服务，切忌不要直接关闭ASM实例，在11g中 11.2.0.2 中 ocr 服务的信息存放在asm 磁盘上，是依赖于asm的，所以要关闭crs 而不是直接关闭asm实例，若在此处强行关闭asm实例，很有可能造成的后果是asm磁盘损坏或者丢失，导致整个集群彻底崩溃。我在测试RAC时，曾dismount某个磁盘组后，首先关闭了数据库，接着直接关闭了asm实例，关闭时一切正常。无报错，但当启动时，发现asm无法启动crs提示CRS-0184: Cannot communicate with the CRS daemon.无论如何也是无法启动CRS的，意识到可能是表决磁盘或者OCR出现问题，随即查看asm磁盘情况：oracleasm listdisks，没有任何磁盘信息，oracleasm scandisks也是同样的情况，这下估计麻烦了。在登录到另外一个节点上，同样命令查看asm磁盘，发现磁盘正常。故此此时我也将该节点的数据库及asm实例关闭，然后重启服务器，之后悲剧的一幕想当然的发生了，两个节点都无法扫描到asm磁盘信息。整个集群破坏的很彻底，丢失数据。也许是我个人原因造成这种异常的数据丢失，但个人还是建议按照常规的顺序来关闭启动RAC。
2. [root@RAC01 bin]# ./crsctl stop crs
如果你足够细心，你可以通过下面的关闭日志看出在RAC的CRS关闭的过程中是一个怎样的顺序以及关闭crs的时候都会有那些服务会同时关闭。此外你可能还会发现我们的这个命令是在RAC01这个节点执行的，所以这时候应该RAC02这个节点还是可以正常使用的。强烈建议大家对下面的日志进行研究。
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac01'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac01'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac01'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'rac01'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rac01'
CRS-2673: Attempting to stop 'ora.FLASH.dg' on 'rac01'
CRS-2673: Attempting to stop 'ora.OCRVD.dg' on 'rac01'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'rac01'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rac01'
CRS-2673: Attempting to stop 'ora.oc4j' on 'rac01'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'rac01'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rac01'
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'rac01' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'rac01'
CRS-2677: Stop of 'ora.scan3.vip' on 'rac01' succeeded
CRS-2672: Attempting to start 'ora.scan3.vip' on 'rac02'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'rac01' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'rac01'
CRS-2677: Stop of 'ora.scan2.vip' on 'rac01' succeeded
CRS-2672: Attempting to start 'ora.scan2.vip' on 'rac02'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rac01' succeeded
CRS-2673: Attempting to stop 'ora.rac01.vip' on 'rac01'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rac01' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rac01'
CRS-2677: Stop of 'ora.rac01.vip' on 'rac01' succeeded
CRS-2672: Attempting to start 'ora.rac01.vip' on 'rac02'
CRS-2677: Stop of 'ora.scan1.vip' on 'rac01' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'rac02'
CRS-2677: Stop of 'ora.registry.acfs' on 'rac01' succeeded
CRS-2677: Stop of 'ora.FLASH.dg' on 'rac01' succeeded
CRS-2676: Start of 'ora.scan3.vip' on 'rac02' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'rac02'
CRS-2676: Start of 'ora.scan2.vip' on 'rac02' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'rac02'
CRS-2676: Start of 'ora.rac01.vip' on 'rac02' succeeded
CRS-2676: Start of 'ora.scan1.vip' on 'rac02' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'rac02'
CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'rac02' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'rac02' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'rac02' succeeded
CRS-2677: Stop of 'ora.OCRVD.dg' on 'rac01' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'rac01' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac01'
CRS-2677: Stop of 'ora.asm' on 'rac01' succeeded
CRS-2677: Stop of 'ora.oc4j' on 'rac01' succeeded
CRS-2672: Attempting to start 'ora.oc4j' on 'rac02'
CRS-2676: Start of 'ora.oc4j' on 'rac02' succeeded
CRS-2673: Attempting to stop 'ora.eons' on 'rac01'
CRS-2673: Attempting to stop 'ora.ons' on 'rac01'
CRS-2677: Stop of 'ora.ons' on 'rac01' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'rac01'
CRS-2677: Stop of 'ora.net1.network' on 'rac01' succeeded
CRS-2677: Stop of 'ora.eons' on 'rac01' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac01' has completed
CRS-2677: Stop of 'ora.crsd' on 'rac01' succeeded
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'rac01'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac01'
CRS-2673: Attempting to stop 'ora.evmd' on 'rac01'
CRS-2673: Attempting to stop 'ora.asm' on 'rac01'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'rac01'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac01'
CRS-2677: Stop of 'ora.cssdmonitor' on 'rac01' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rac01' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'rac01' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'rac01' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'rac01' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac01' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac01'
CRS-2677: Stop of 'ora.cssd' on 'rac01' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac01'
CRS-2673: Attempting to stop 'ora.diskmon' on 'rac01'
CRS-2677: Stop of 'ora.gpnpd' on 'rac01' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac01'
CRS-2677: Stop of 'ora.gipcd' on 'rac01' succeeded
CRS-2677: Stop of 'ora.diskmon' on 'rac01' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac01' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@RAC01 bin]#
3. 验证关闭情况：
[root@RAC01 bin]# ./crs_stat -t -v
CRS-0184: Cannot communicate with the CRS daemon.
[root@RAC01 bin]#
4. 启动crs服务：
[root@RAC01 bin]# ./crsctl start crs-------------此处可能需要等待一段时间。
CRS-4123: Oracle High Availability Services has been started.
[root@RAC01 bin]#
5. 查看启动情况：
[grid@RAC02 ~]$ crs_stat -t -v
Name Type R/RA F/FT Target State Host
----------------------------------------------------------------------
ora.DATA.dg ora....up.type 0/5 0/ ONLINE ONLINE rac02
ora.FLASH.dg ora....up.type 0/5 0/ ONLINE ONLINE rac02
ora....ER.lsnr ora....er.type 0/5 0/ ONLINE ONLINE rac01
ora....N1.lsnr ora....er.type 0/5 0/0 ONLINE ONLINE rac01
ora....N2.lsnr ora....er.type 0/5 0/0 ONLINE ONLINE rac02
ora....N3.lsnr ora....er.type 0/5 0/0 ONLINE ONLINE rac02
ora....N4.lsnr ora....er.type 0/5 0/0 ONLINE ONLINE rac02
ora.OCRVD.dg ora....up.type 0/5 0/ ONLINE ONLINE rac01
ora.asm ora.asm.type 0/5 0/ ONLINE ONLINE rac01
ora.eons ora.eons.type 0/3 0/ ONLINE ONLINE rac01
ora.gsd ora.gsd.type 0/5 0/ OFFLINE OFFLINE
ora....network ora....rk.type 0/5 0/ ONLINE ONLINE rac01
ora.oc4j ora.oc4j.type 0/5 0/0 ONLINE ONLINE rac02
ora.ons ora.ons.type 0/3 0/ ONLINE ONLINE rac01
ora....SM1.asm application 0/5 0/0 ONLINE ONLINE rac01
ora....01.lsnr application 0/5 0/0 ONLINE ONLINE rac01
ora.rac01.gsd application 0/5 0/0 OFFLINE OFFLINE
ora.rac01.ons application 0/3 0/0 ONLINE ONLINE rac01
ora.rac01.vip ora....t1.type 0/0 0/0 ONLINE ONLINE rac01
ora....SM2.asm application 0/5 0/0 ONLINE ONLINE rac02
ora....02.lsnr application 0/5 0/0 ONLINE ONLINE rac02
ora.rac02.gsd application 0/5 0/0 OFFLINE OFFLINE
ora.rac02.ons application 0/3 0/0 ONLINE ONLINE rac02
ora.rac02.vip ora....t1.type 0/0 0/0 ONLINE ONLINE rac02
ora.racwh.db ora....se.type 0/2 0/1 OFFLINE OFFLINE
ora....f01.svc ora....ce.type 0/0 1/0 OFFLINE OFFLINE-----------我个人添加的STAF服务。
ora....f02.svc ora....ce.type 0/0 0/0 OFFLINE OFFLINE-----------我个人添加的STAF服务。
ora....ry.acfs ora....fs.type 0/5 0/ ONLINE ONLINE rac01
ora.scan1.vip ora....ip.type 0/0 1/0 ONLINE ONLINE rac01
ora.scan2.vip ora....ip.type 0/0 1/0 ONLINE ONLINE rac02
ora.scan3.vip ora....ip.type 0/0 0/0 ONLINE ONLINE rac02
ora.scan4.vip ora....ip.type 0/0 0/0 ONLINE ONLINE rac02
[grid@RAC02 ~]$
6. 启动数据库：
[grid@RAC02 ~]$ srvctl start database -d RACwh
PRCR-1079 : Failed to start resource ora.racwh.db
ORA-16038: log 3 sequence# 41 cannot be archived
ORA-19504: failed to create file ""
ORA-00312: online log 3 thread 2: '+DATA/racwh/redo03.log'
CRS-2674: Start of 'ora.racwh.db' on 'rac01' failed
CRS-2632: There are no more servers to try to place resource 'ora.racwh.db' on that would satisfy its placement policy
[grid@RAC02 ~]$
针对上述的报错，经查看原来是归档日志所在的asm未能mount所致，执行该磁盘的mount后重新执行数据库启动即可。
此外给大家推荐几个日常的启动关闭命令：
其他管理维护指南
一.通过srvctl管理
1.关闭RAC数据库服务顺序
[oracle@dbp ~] $ srvctl stop database -d RACwh
[oracle@dbp ~] $ srvctl stop asm -n RAC01
[oracle@dbp ~] $ srvctl stop asm -n RAC02
[oracle@dbp ~] $ srvctl stop nodeapps -n RAC01
[oracle@dbp ~] $ srvctl stop nodeapps -n RAC02
2.启动RAC数据库服务顺序
[oracle@RAC01 ~] $ srvctl start nodeapps -n RAC01
[oracle@RAC01 ~] $ srvctl start nodeapps -n RAC02
[oracle@RAC01 ~] $ srvctl start asm -n RAC01
[oracle@RAC01 ~] $ srvctl start asm -n RAC02
[oracle@RAC01 ~] $ srvctl start database -d RACwh
3.其他命令
实例管理
[oracle@RAC01 ~] $ srvctl status instance -d RACwh -i RACwh2
[oracle@RAC01 ~] $ srvctl stop instance -d RACwh -i RACwh2
[oracle@RAC01 ~] $ srvctl start instance -d RACwh -i RACwh2
监听管理
[oracle@RAC01 ~] $ srvctl status listener -n RAC01
[oracle@RAC01 ~] $ srvctl stop listener -n RAC01
[oracle@RAC01 ~] $ srvctl start listener -n RAC01
二.通过crsctl工具管理
[oracle@RAC01 ~] $ crs_stat -t
[oracle@RAC01 ~] $ crs_stat
[oracle@RAC01 ~] $ crs_start -all
[oracle@RAC01 ~] $ crs_stop -all
[oracle@RAC01 ~] $ crs_stop "ora. RACwh.db"
Crs_stat命令在oracle11g后逐渐趋于废弃，如下即可明白：
[grid@RAC02 ~]$ crs_stat -h
This command is deprecated and has been replaced by 'crsctl status resource'
This command remains for backward compatibility only
Usage: crs_stat [resource_name [...]] [-v] [-l] [-q] [-c cluster_member]
crs_stat [resource_name [...]] -t [-v] [-q] [-c cluster_member]
crs_stat -p [resource_name [...]] [-q]
crs_stat [-a] application -g
crs_stat [-a] application -r [-c cluster_member]
crs_stat -f [resource_name [...]] [-q] [-c cluster_member]
crs_stat -ls [resource_name [...]] [-q]
