EMagent 12c with high number of threads causing “su: cannot set user id: Resource temporarily unavailable”

Hello all,
Yeah, these days I got some errors and when validating the server found the fllowing error:

su: cannot set user id: Resource temporarily unavailable

As you can imagine, in order to fix the issue, I adjusted the /etc/security/limits.conf increasing oracle nprocs to:

oracle           soft    nproc           4047
oracle           hard    nproc           20384

Ok, turns out that after a while I got the same errors again…
After some investigating I find that the EM Agent process was with 5020 threads!
Take a look:

proddb $: su - oracle
su: cannot set user id: Resource temporarily unavailable
proddb $: ps -LF -U oracle | wc -l
proddb $: ps -ef |grep agent
oracle   20051 19957 20051  0 5020 1505103 1001352 6 2016 ?       00:00:00 /u01/app/oracle/agent/core/ -Xmx128M -XX:MaxPermSize=96M -server -Djava.security.egd=file:///dev/./urandom -Dsun.lang.ClassLoader.allowArraySyntax=true -XX:+UseLinuxPosixThreadCPUClocks -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+UseCompressedOops -Dwatchdog.pid=19957 -cp /u01/app/oracle/agent/core/ oracle.sysman.gcagent.tmmain.TMMain
proddb $: grep 20051 /tmp/boesing | wc -l

First actions was to KILL em agent process and restart agent:

proddb $: kill -9 20051
proddb $: su - oracle
oracle $: . oraenv
ORACLE_SID = [] ? emagent
You can set ORACLE_BASE manually if it is required.
oracle $: emctl start agent
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Starting agent ................ started.
oracle $: ps -LF -U oracle | wc -l

Way better!
After this, and some investigation, the summary this behavior is a match to MOS Bug 21459792: EM Agent creating many threads.

This bug have no solution/fix, only a workaround: Restart the agent after kill the agent process.

Hmmm not good, right?

This is not official, but there is a very nice post by Franck Pachot about the subject.
He found some relation with resource ora.eons. In his post, a possible action to fix is to disable EM Agent subscription to this resource:

emctl setproperty agent -name disableEonsRcvlet -value true

I’d also recommend you to check this doc if you are having some similar errors:
-EM12C: Troubleshooting Guide for High Memory Consumption by EM 12c Agent(Doc ID 1556582.1)

Hope it helps you…

See you next week!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s