EMagent 12c with high number of threads causing “su: cannot set user id: Resource temporarily unavailable”

Hello all,
Yeah, these days I got some errors and when validating the server found the fllowing error:

su: cannot set user id: Resource temporarily unavailable

As you can imagine, in order to fix the issue, I adjusted the /etc/security/limits.conf increasing oracle nprocs to:

oracle           soft    nproc           4047
oracle           hard    nproc           20384

Ok, turns out that after a while I got the same errors again…
After some investigating I find that the EM Agent process was with 5020 threads!
Take a look:

proddb $: su - oracle
su: cannot set user id: Resource temporarily unavailable
proddb $: ps -LF -U oracle | wc -l
5377
proddb $: ps -ef |grep agent
oracle   20051 19957 20051  0 5020 1505103 1001352 6 2016 ?       00:00:00 /u01/app/oracle/agent/core/12.1.0.3.0/jdk/bin/java -Xmx128M -XX:MaxPermSize=96M -server -Djava.security.egd=file:///dev/./urandom -Dsun.lang.ClassLoader.allowArraySyntax=true -XX:+UseLinuxPosixThreadCPUClocks -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+UseCompressedOops -Dwatchdog.pid=19957 -cp /u01/app/oracle/agent/core/12.1.0.3.0/jdbc/lib/ojdbc5.jar:/u01/app/oracle/agent/core/12.1.0.3.0/ucp/lib/ucp.jar:/u01/app/oracle/agent/core/12.1.0.3.0/modules/oracle.http_client_11.1.1.jar:/u01/app/oracle/agent/core/12.1.0.3.0/lib/xmlparserv2.jar:/u01/app/oracle/agent/core/12.1.0.3.0/lib/jsch.jar:/u01/app/oracle/agent/core/12.1.0.3.0/lib/optic.jar:/u01/app/oracle/agent/core/12.1.0.3.0/modules/oracle.dms_11.1.1/dms.jar:/u01/app/oracle/agent/core/12.1.0.3.0/modules/oracle.odl_11.1.1/ojdl.jar:/u01/app/oracle/agent/core/12.1.0.3.0/modules/oracle.odl_11.1.1/ojdl2.jar:/u01/app/oracle/agent/core/12.1.0.3.0/sysman/jlib/log4j-core.jar:/u01/app/oracle/agent/core/12.1.0.3.0/jlib/gcagent_core.jar:/u01/app/oracle/agent/core/12.1.0.3.0/sysman/jlib/emagentSDK-intg.jar:/u01/app/oracle/agent/core/12.1.0.3.0/sysman/jlib/emagentSDK.jar oracle.sysman.gcagent.tmmain.TMMain
proddb $: grep 20051 /tmp/boesing | wc -l
5020

First actions was to KILL em agent process and restart agent:

proddb $: kill -9 20051
proddb $: su - oracle
oracle $: . oraenv
ORACLE_SID = [] ? emagent
You can set ORACLE_BASE manually if it is required.
oracle $: emctl start agent
Oracle Enterprise Manager Cloud Control 12c Release 3
Copyright (c) 1996, 2013 Oracle Corporation.  All rights reserved.
Starting agent ................ started.
oracle $: ps -LF -U oracle | wc -l
472

Way better!
After this, and some investigation, the summary this behavior is a match to MOS Bug 21459792: EM 12.1.0.3 Agent creating many threads.

This bug have no solution/fix, only a workaround: Restart the agent after kill the agent process.

Hmmm not good, right?

This is not official, but there is a very nice post by Franck Pachot about the subject.
He found some relation with resource ora.eons. In his post, a possible action to fix is to disable EM Agent subscription to this resource:

emctl setproperty agent -name disableEonsRcvlet -value true

I’d also recommend you to check this doc if you are having some similar errors:
-EM12C: Troubleshooting Guide for High Memory Consumption by EM 12c Agent(Doc ID 1556582.1)

Hope it helps you…

See you next week!