Exadata Compute Node Not Starting after a long Period…

Well,
After a long time on a graceful reboot, the compute node was simply not starting… What do to?
The best is:

1. Connect to ILOM Console:

Go to: Host Management –> Power control –> select Power Cycle in drop down list.

2. Connect to ILOM Server start SP console:
You may do it from another node, of course.

[root@grepora02 ~]# ssh root@grepora01-ilom Password:

Oracle(R) Integrated Lights Out Manager

Version 3.2.9.23 r116695

Copyright (c) 2017, Oracle and/or its affiliates. All rights reserved.

Warning: HTTPS certificate is set to factory default.

Hostname: grepora01-ilom

-> start /SP/console Are you sure you want to start /SP/console (y/n)? y

And, if not, as always, create a SR and follow with Oracle is the best way to go…

Hope it helps!

Services [not] Starting Automatically with CRS after Reboot

Hello all!
So, these days a client asked me to check why his database was not starting with CRS after reboot. So I started investigating and noticed this…

On Oracle 11.2, the database auto start policy in the clusterware is restore, which means that clusterware will remember the last state of the database. As well as database, Oracle 11.2 comes by default with several important resources with attribute AUTO_START=restore in the profile.

With that, if the database was stopped normally then on the next restart of clusterware it won’t be started. Otherwise if the server crashes or by some reason the OS is rebooted then clusterware will start the database because last state was ONLINE (running).

In my case, a second reboot was done with database down, and this is why database didn’t started. And same applied to some other target types.

The conclusion I get?
For best availability, it is a good practice to change this default behaviour after each installation on 11.2. Some checking and adjusting scripts below:

To check all Resources:

crsctl stat res -p

Checking on database resource:

...
NAME=ora.grepora.db
TYPE=ora.database.type
ACL=owner:oracle:rwx,pgrp:oinstall:rwx,other::r--
ACTION_FAILURE_TEMPLATE=
ACTION_SCRIPT=
ACTIVE_PLACEMENT=1
AGENT_FILENAME=%CRS_HOME%/bin/oraagent%CRS_EXE_SUFFIX%
AUTO_START=restore
...

But besides DB, several resources were in restore or even in never, like ASM and Voting Disk DG… Check for all resourced AUTO_START:

[root@greporasrv1 ~]# crsctl stat res -p |grep AUTO_START
AUTO_START=always
AUTO_START=always
AUTO_START=always
AUTO_START=restore
AUTO_START=restore
AUTO_START=restore
AUTO_START=restore
AUTO_START=never
AUTO_START=never
AUTO_START=restore
AUTO_START=always
AUTO_START=restore
AUTO_START=restore
AUTO_START=restore
AUTO_START=restore
AUTO_START=restore
AUTO_START=always
AUTO_START=restore
AUTO_START=restore
AUTO_START=restore
AUTO_START=restore

Well, so to fix all the resources:

crsctl modify resource ora.APS1.dg -attr AUTO_START=always
crsctl modify resource ora.DG1.dg -attr AUTO_START=always
crsctl modify resource ora.FRA.dg -attr AUTO_START=always
crsctl modify resource ora.LISTENER.lsnr -attr AUTO_START=always
crsctl modify resource ora.VDISK_OCR.dg -attr AUTO_START=always
crsctl modify resource ora.asm -attr AUTO_START=always
crsctl modify resource ora.gsd -attr AUTO_START=always
crsctl modify resource ora.net1.network -attr AUTO_START=always
crsctl modify resource ora.ons -attr AUTO_START=always
crsctl modify resource ora.registry.acfs -attr AUTO_START=always
crsctl modify resource ora.LISTENER_SCAN1.lsnr -attr AUTO_START=always
crsctl modify resource ora.LISTENER_SCAN2.lsnr -attr AUTO_START=always
crsctl modify resource ora.LISTENER_SCAN3.lsnr -attr AUTO_START=always
crsctl modify resource ora.cvu -attr AUTO_START=always
crsctl modify resource ora.grepora.db -attr AUTO_START=always
crsctl modify resource ora.greporasrv1.vip -attr AUTO_START=always
crsctl modify resource ora.greporasrv2.vip -attr AUTO_START=always
crsctl modify resource ora.oc4j -attr AUTO_START=always
crsctl modify resource ora.scan1.vip -attr AUTO_START=always
crsctl modify resource ora.scan2.vip -attr AUTO_START=always
crsctl modify resource ora.scan3.vip -attr AUTO_START=always

Below the output and the checking:

[root@greporasrv1 ~]# crsctl modify resource ora.LISTENER.lsnr -attr AUTO_START=always
[root@greporasrv1 ~]# crsctl modify resource ora.VDISK_OCR.dg -attr AUTO_START=always
[root@greporasrv1 ~]# crsctl modify resource ora.asm -attr AUTO_START=always
[root@greporasrv1 ~]# crsctl modify resource ora.gsd -attr AUTO_START=always
[root@greporasrv1 ~]# crsctl modify resource ora.net1.network -attr AUTO_START=always
[root@greporasrv1 ~]# crsctl modify resource ora.ons -attr AUTO_START=always
[root@greporasrv1 ~]# crsctl modify resource ora.registry.acfs -attr AUTO_START=always
[root@greporasrv1 ~]# crsctl modify resource ora.LISTENER_SCAN1.lsnr -attr AUTO_START=always
[root@greporasrv1 ~]# crsctl modify resource ora.LISTENER_SCAN2.lsnr -attr AUTO_START=always
[root@greporasrv1 ~]# crsctl modify resource ora.LISTENER_SCAN3.lsnr -attr AUTO_START=always
[root@greporasrv1 ~]# crsctl modify resource ora.cvu -attr AUTO_START=always
[root@greporasrv1 ~]# crsctl modify resource ora.grepora.db -attr AUTO_START=always
[root@greporasrv1 ~]# crsctl modify resource ora.greporasrv1.vip -attr AUTO_START=always
[root@greporasrv1 ~]# crsctl modify resource ora.greporasrv2.vip -attr AUTO_START=always
[root@greporasrv1 ~]# crsctl modify resource ora.oc4j -attr AUTO_START=always
[root@greporasrv1 ~]# crsctl modify resource ora.scan1.vip -attr AUTO_START=always
[root@greporasrv1 ~]# crsctl modify resource ora.scan2.vip -attr AUTO_START=always
[root@greporasrv1 ~]# crsctl modify resource ora.scan3.vip -attr AUTO_START=always
[root@greporasrv1 ~]#
[root@greporasrv1 ~]# crsctl stat res -p |grep AUTO_START
AUTO_START=always
AUTO_START=always
AUTO_START=always
AUTO_START=always
AUTO_START=always
AUTO_START=always
AUTO_START=always
AUTO_START=always
AUTO_START=always
AUTO_START=always
AUTO_START=always
AUTO_START=always
AUTO_START=always
AUTO_START=always
AUTO_START=always
AUTO_START=always
AUTO_START=always
AUTO_START=always
AUTO_START=always
AUTO_START=always
AUTO_START=always
[root@greporasrv1 ~]#