PRCR-1079 CRS-2674 CRS-5017 ORA-27102: out of memory Linux-x86_64 Error: 28: No space left on device

# Problem

myserver:/home/oracle>srvctl start database -d mydb
PRCR-1079 : Failed to start resource ora.mydb.db
CRS-5017: The resource action "ora.mydb.db start" encountered the following error:
ORA-27102: out of memory
Linux-x86_64 Error: 28: No space left on device
. For details refer to "(:CLSN00107:)" in "/grid/product/11.2.0/log/myserver/agent/crsd/oraagent_oracle/oraagent_oracle.log".
CRS-2674: Start of 'ora.mydb.db' on 'myserver' failed
CRS-5017: The resource action "ora.mydb.db start" encountered the following error:
ORA-27102: out of memory
Linux-x86_64 Error: 28: No space left on device
. For details refer to "(:CLSN00107:)" in "/grid/product/11.2.0/log/myserver2/agent/crsd/oraagent_oracle/oraagent_oracle.log".
CRS-2674: Start of 'ora.mydb.db' on 'myserver2' failed
CRS-2632: There are no more servers to try to place resource 'ora.mydb.db' on that would satisfy its placement policy
myserver:/home/oracle>

# Solution

On /etc/sysctl.conf ajust as below and then reload sysctl (“sysctl -p” as root):

#Old
#kernel.shmall = 24641536
#New
kernel.shmall = 4294967296

Matheus.

Is My Linux Server Physical or Virtual?

Supposing you are in a server shell and don’t know if you machine is virtualized (a VM)?
One way to check that (supposing VMWare as hypervisioning solution) is:

[root@mydbsrvr ~]# dmidecode | grep -i vmware
Manufacturer: VMware, Inc.
Product Name: VMware Virtual Platform
Serial Number: VMware-xx xx xx xx xx xx xx xx-xx xx xx xx xx xx xx xx
Description: VMware SVGA II

If you had an answer like this, yes, it’s a VM. 🙂

Matheus.

ORA-27369: job of type EXECUTABLE failed with exit code: Operation not permitted

When running external script by scheduler. The solution:

chown root $ORACLE_HOME/bin/extjob
chmod 4750 $ORACLE_HOME/bin/extjob

chown root $ORACLE_HOME/rdbms/admin/externaljob.ora
chmod 640 $ORACLE_HOME/rdbms/admin/externaljob.ora

chown root $ORACLE_HOME/bin/jssu
chmod 4750 $ORACLE_HOME/bin/jssu

 

Have a nice week!
Matheus.

Mount Diretory from Remote RHEL7 Server (NFS)

Quick Post: To mount a directory via NFS from a RHEL7 remote server:

Souce Host:

[root@sourcehost ~]# cat /etc/exports
/oracle/sharedir targethost(rw,no_root_squash,insecure)
[root@sourcehost ~]# /bin/systemctl restart nfs.service

* Note: The “/bin/systemctl” is the new by RHEL7. For other versions you can just use “service nfs restart”.

Target Host:

[root@targethost ~]# mkdir -p /sourcehost/sharedir
[root@targethost ~]# mount -t nfs sourcehost:/oracle/sharedir /sourcehost/sharedir
[root@srac-his ~]# df -h /sourcehost/sharedir
Filesystem Size Used Avail Use% Mounted on
sourcehost:/oracle/sharedir
100G 279M 100G 1% /sourcehost/sharedir

 

Have a nice weekend!
Matheus.

VKTM Hang – High CPU Usage

Today a database (RHEL 6, single instance, 11.2.0.4) suddently started to “explode” CPU on VKTM process (100% CPU).
After some minutes lost (completely) in support.oracle.com (there was just a few notes about binary permissions on Solaris), I decided to make a McGayver by myself. 🙂

By Oracle words: “VKTM acts as a time publisher for an Oracle instance. VKTM publishes two sets of time: a wall clock time using a seconds interval and a higher resolution time (which is not wall clock time) for interval measurements. The VKTM timer service centralizes time tracking and offloads multiple timer calls from other clients.

This way, my solution:

SQL> alter system set "_high_priority_processes"='LMS*' scope=spfile;
System altered.

And restart the database, of course.
So, VKTM is no more a “priority” process. The problem was solved. 🙂

Another possibility is to disable VKTM (undocumented parameter “_disable_vktm” – boolean). But I wanted to keep it running, changing less as possible of database configuration, just reducing priority.

KB:
Master Note: Troubleshooting Oracle Background Processes (Doc ID 1509616.1)
Great post about hidden parameters: http://oracleinaction.com/undocumented-params-11g/
Oficial one: http://www.orafaq.com/parms/index.htm

Hugs!
Matheus.

Kludge: Mounting ACFS Thought Shellscript

Just the script. The history is here.
This is a “workaround” script. As always, is recommended to use Oracle Restart, like I posted here.

#!/bin/sh
$GRID_HOME/bin/srvctl add filesystem -d /dev/asm/dbhome-270 -g 'DGHOME' -v DBHOME -m /oracle/db -u oracle
if [ $? = "0" -o $? = "2" ]; then
$GRID_HOME/bin/srvctl start filesystem -d /dev/asm/dbhome-270
if [ $? = "0" ]; then
chown oracle:oinstall /oracle/db
chmod 775 /oracle/db
$GRID_HOME/bin/srvctl status filesystem -d /dev/asm/dbhome-270
exit 0
fi
$GRID_HOME/bin/srvctl status filesystem -d /dev/asm/dbhome-270
fi

There is a good post ACFS and ACFS restart scripting:
https://levipereira.wordpress.com/2011/07/28/oracle-acfs-filesystem-managed-by-ohas-on-oracle-restart/

See ya!

Matheus.

Flush DNS on Linux

I began posting about ORA-12514 after database migration involving DNS adjustment.
Then, to make it more clear I wrote about How to Flush DNS Cache.

Now, just a complementar information that can be usefull:

# To invalidade DNS Cache:

ls /var/db/nscd/
group hosts netgroup passwd services
 
nscd --invalidate=hosts  (or -i hosts)

Hugs!

Matheus.

ORA-15186: ASMLIB error function

Almost a month away… My bad!
Here I go again, with a quick tip, that a passed today. Our kernel was ‘changed’ without advise and this began to happen:

ORA-15186: ASMLIB error function = [asm_init], error = [18446744073709551611], mesg = [Driver not installed]
ERROR: error ORA-15186 caught in ASM I/O path

The solution was is basically update the asmlibs, that is based on kernel version. For RHEL, the solution is well decribed here:

http://www.oracle.com/technetwork/server-storage/linux/asmlib/rhel6-1940776.html
https://access.redhat.com/solutions/315643

Just to remember: After the kernel change, a relink of your Oracle Home is higly recommended.

Have a nice day!
Matheus.