Hi all,
So I got this error from OEM in a client. When connecting noticed OEM agent was down.
From logs, it failed to restart because of error “java.lang.OutOfMemoryError: Java heap space“.
I tried to run clearstate but got same error.
Searched on MOS – found MOS Duplicate 1952593.1: EM12c: emctl start agent Fails With Target Interaction Manager failed at Startup java.lang.OutOfMemoryError: Java heap space reported in gcagent_errors.log (Doc ID 1902124.1)
Solution seems to be moving /agent_inst/sysman/emd/state/ content to another location before running clearstate.
After doing so, all worked fine! So if you are facing the same, try this out!
oracle@dbserver:/db/oracle/product/agent12c/12.1.0.4/agent_inst/bin$ ./emctl status agent Oracle Enterprise Manager Cloud Control 12c Release 4 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved. --------------------------------------------------------------- Agent is Not Running oracle@dbserver:/db/oracle/product/agent12c/12.1.0.4/agent_inst/bin$ cd .. oracle@dbserver:/db/oracle/product/agent12c/12.1.0.4/agent_inst$ cd sysman oracle@dbserver:/db/oracle/product/agent12c/12.1.0.4/agent_inst/sysman$ cd log/ oracle@dbserver:/db/oracle/product/agent12c/12.1.0.4/agent_inst/sysman/log$ ls -lrt | tail -rw-r----- 1 oracle dba 4096 Jan 9 09:47 emagent.nohup.lr -rw-r----- 1 oracle dba 11165 Jan 9 09:48 startup.info -rw-r----- 1 oracle dba 153432 Jan 9 09:48 gcagent_errors.log -rw------- 1 oracle dba 540010788 Jan 9 09:48 heapDump_7.hprof -rw-r----- 1 oracle dba 4687452 Jan 9 09:48 gcagent.log -rw-r----- 1 oracle dba 1746 Jan 9 09:48 agabend.log -rw-r----- 1 oracle dba 2863 Jan 9 09:48 gcagent_pfu.log -rw-r----- 1 oracle dba 84308 Jan 9 09:48 emagent.nohup -rw-r----- 1 oracle dba 762142 Jan 9 10:06 emdctlj.log -rw-r----- 1 oracle dba 121782 Jan 9 10:06 emctl.log oracle@dbserver:/db/oracle/product/agent12c/12.1.0.4/agent_inst/sysman/log$ date Monday, January 9, 2020 10:06:24 AM PST oracle@dbserver:/db/oracle/product/agent12c/12.1.0.4/agent_inst/sysman/log$ tail gcagent.log 2020-01-09 09:48:50,875 [32:F9C26A76] INFO - *jetty*: Graceful shutdown SslSelectChannelConnector@0.0.0.0:3872 2020-01-09 09:48:50,877 [32:F9C26A76] INFO - *jetty*: Graceful shutdown ContextHandler@63b5a40a@63b5a40a/emd/lifecycle/main,null 2020-01-09 09:48:50,877 [32:F9C26A76] INFO - *jetty*: Graceful shutdown HTTPLifecycleHandler@7284aa02 2020-01-09 09:48:50,877 [32:F9C26A76] INFO - *jetty*: Graceful shutdown ContextHandler@5dac13d7@5dac13d7/emd/main,null 2020-01-09 09:48:50,877 [32:F9C26A76] INFO - *jetty*: Graceful shutdown HTTPRequestHandler@52a34783 2020-01-09 09:48:50,877 [32:F9C26A76] INFO - *jetty*: Graceful shutdown ServletContextHandler@201d592a@201d592a/emd/browser,null 2020-01-09 09:48:50,878 [32:F9C26A76] INFO - *jetty*: Graceful shutdown ContextHandler@19a9bea3@19a9bea3/emd/persistence/main,null 2020-01-09 09:48:50,878 [32:F9C26A76] INFO - *jetty*: Graceful shutdown HTTPAgentPersistenceHandler@3d89acb5 2020-01-09 09:48:50,878 [32:F9C26A76] INFO - *jetty*: Graceful shutdown ContextHandler@5722cc7e@5722cc7e/,null 2020-01-09 09:48:53,932 [32:F9C26A76] INFO - *jetty*: Shutdown hook complete oracle@dbserver:/db/oracle/product/agent12c/12.1.0.4/agent_inst/sysman/log$ tail gcagent_errors.log at oracle.sysman.gcagent.target.interaction.execution.Threshold.initThreshold(Threshold.java:4132) at oracle.sysman.gcagent.target.interaction.execution.TargetInteractionMgr.init(TargetInteractionMgr.java:225) at oracle.sysman.gcagent.target.interaction.execution.TargetInteractionMgr.tmNotifier(TargetInteractionMgr.java:171) at oracle.sysman.gcagent.tmmain.lifecycle.TMComponentSvc.invokeNotifier(TMComponentSvc.java:998) at oracle.sysman.gcagent.tmmain.lifecycle.TMComponentSvc.invokeInitializationStep(TMComponentSvc.java:1083) at oracle.sysman.gcagent.tmmain.lifecycle.TMComponentSvc.doInitializationStep(TMComponentSvc.java:916) at oracle.sysman.gcagent.tmmain.lifecycle.TMComponentSvc.notifierDriver(TMComponentSvc.java:812) at oracle.sysman.gcagent.tmmain.TMMain.startup(TMMain.java:256) at oracle.sysman.gcagent.tmmain.TMMain.agentMain(TMMain.java:557) at oracle.sysman.gcagent.tmmain.TMMain.main(TMMain.java:546) oracle@dbserver:/db/oracle/product/agent12c/12.1.0.4/agent_inst/sysman/log$ less gcagent_errors.log oracle@dbserver:/db/oracle/product/agent12c/12.1.0.4/agent_inst/sysman/log$ cd .. oracle@dbserver:/db/oracle/product/agent12c/12.1.0.4/agent_inst/sysman$ cd config/ oracle@dbserver:/db/oracle/product/agent12c/12.1.0.4/agent_inst/sysman/config$ ls -lrt | tail -rw-r----- 1 oracle dba 8001 May 13 2016 b64LocalCertificate.txt -rw-r----- 1 oracle dba 17893 May 13 2016 b64InternetCertificate.txt -rw-r----- 1 oracle dba 7833 May 13 2016 emd.properties.2016_05_13_10_05_31 -rw------- 1 oracle dba 499 May 20 2016 s_jvm_options.opt.save -rw-r----- 1 oracle dba 8202 Sep 26 01:20 emd.properties.bkp -rw-r----- 1 oracle dba 8204 Jan 9 09:46 emd.properties.bak -rw-r----- 1 oracle dba 156 Jan 9 09:46 private.properties.bak -rw-r----- 1 oracle dba 8204 Jan 9 09:48 emd.properties -rw-r----- 1 oracle dba 266 Jan 9 09:48 autotune.properties -rw-r----- 1 oracle dba 156 Jan 9 09:48 private.properties oracle@dbserver:/db/oracle/product/agent12c/12.1.0.4/agent_inst/sysman/config$ date Monday, January 9, 2020 10:07:56 AM PST oracle@dbserver:/db/oracle/product/agent12c/12.1.0.4/agent_inst/sysman/config$ n/emctl status agent Oracle Enterprise Manager Cloud Control 12c Release 4 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved. --------------------------------------------------------------- Agent is Not Running oracle@dbserver:/db/oracle/product/agent12c/12.1.0.4/agent_inst/sysman/config$ which emctl /db/oracle/product/agent12c/12.1.0.4/agent_inst/bin/emctl oracle@dbserver:/db/oracle/product/agent12c/12.1.0.4/agent_inst/sysman/config$ e agent; emctl start agent Oracle Enterprise Manager Cloud Control 12c Release 4 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved. EMD clearstate failed: Offline clearstate failed : java.lang.OutOfMemoryError: Java heap space Oracle Enterprise Manager Cloud Control 12c Release 4 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved. Starting agent ...........................................................................................................................failed. Consult emctl.log and emagent.nohup in: /db/oracle/product/agent12c/12.1.0.4/agent_inst/sysman/log oracle@dbserver:/db/oracle/product/agent12c/12.1.0.4/agent_inst/sysman/config$ oracle@dbserver:/db/oracle/product/agent12c/12.1.0.4/agent_inst/sysman/config$ cd .. oracle@dbserver:/db/oracle/product/agent12c/12.1.0.4/agent_inst/sysman$ cd emd oracle@dbserver:/db/oracle/product/agent12c/12.1.0.4/agent_inst/sysman/emd$ mkdir state-20200109 oracle@dbserver:/db/oracle/product/agent12c/12.1.0.4/agent_inst/sysman/emd$ mv state/* state\-20200109/ oracle@dbserver:/db/oracle/product/agent12c/12.1.0.4/agent_inst/sysman/emd$ du -hs state* 1K state 166M state-20200109 oracle@dbserver:/db/oracle/product/agent12c/12.1.0.4/agent_inst/sysman/emd$ emctl clearstate agent; emctl sta Oracle Enterprise Manager Cloud Control 12c Release 4 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved. EMD clearstate completed successfully Oracle Enterprise Manager Cloud Control 12c Release 4 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved. Starting agent ......................... started. oracle@dbserver:/db/oracle/product/agent12c/12.1.0.4/agent_inst/sysman/emd$ oracle@dbserver:/db/oracle/product/agent12c/12.1.0.4/agent_inst/sysman/emd$ emctl status agent Oracle Enterprise Manager Cloud Control 12c Release 4 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved. --------------------------------------------------------------- Agent Version : 12.1.0.4.0 OMS Version : 12.1.0.4.0 Protocol Version : 12.1.0.1.0 Agent Home : /db/oracle/product/agent12c/12.1.0.4/agent_inst Agent Log Directory : /db/oracle/product/agent12c/12.1.0.4/agent_inst/sysman/log Agent Binaries : /db/oracle/product/agent12c/12.1.0.4/core/12.1.0.4.0 Agent Process ID : 27885 Parent Process ID : 27848 Agent URL : https://x0319vp114.nordstrom.net:3872/emd/main/ Local Agent URL in NAT : https://x0319vp114.nordstrom.net:3872/emd/main/ Repository URL : https://oemcloud.nordstrom.net:4900/empbs/upload Started at : 2020-01-09 10:15:39 Started by user : oracle Operating System : SunOS version 5.11 (sparcv9) Last Reload : (none) Last successful upload : 2020-01-09 10:17:35 Last attempted upload : 2020-01-09 10:17:35 Total Megabytes of XML files uploaded so far : 0.59 Number of XML files pending upload : 1 Size of XML files pending upload(MB) : 0 Available disk space on upload filesystem : 50.38% Collection Status : Collections enabled Heartbeat Status : Ok Last attempted heartbeat to OMS : 2020-01-09 10:16:54 Last successful heartbeat to OMS : 2020-01-09 10:16:54 Next scheduled heartbeat to OMS : 2020-01-09 10:17:54 --------------------------------------------------------------- Agent is Running and Ready oracle@dbserver:/db/oracle/product/agent12c/12.1.0.4/agent_inst/sysman/emd$
Wat did you actually do. From your caommands you tail a bunch of files and started the agent?