Month: August 2017

Enterprise Manager

Oracle ASR: Communication Issues

 , , ,

Hello all,
Are you having notifications like this one from you ASR?

ALERT: Oracle Auto Service Request (ASR) has detected a heartbeat failure for these assets.

[list with "Serial"; "Hostnam"; "Information" of affected targets]

IMPACT: ASR would not be able to create a Service Request (SR) if a fault were to occur.
ACTION: Determine why the heartbeat has failed for these assets and resolve the issue.

This is only a notification saying that ASR not able to reach a target.
For detailed information on how to troubleshoot, you can access MOS Oracle Auto Service Request (ASR) No Heartbeat Issue – How to Resolve (Doc ID 1346328.2)

In general why, things to test are:
– Access from ASR server to transport.oracle.com, via https, using port 443.
– Access from ASR server to ASR assets, via http, using port 6481.

More“Oracle ASR: Communication Issues”

Application | Middleware | GoldenGate, Fusion Middleware

Failed to invoke startup class “JRF Startup Class” in OSB or SOA.

 , , , , ,

Hi guys!!!

After installation of soa-infra or osb, in the first time the the managed server starts, the error occurs:

####     <[ACTIVE] ExecuteThread: '0' for queue: 'weblo gic.kernel.Default (self-tuning)'> <> <> <> <1478002511204>  
at weblogic.management.deploy.classdeployment.ClassDeploymentManager.invokeClass(ClassDeploymentManager.java:274)
at weblogic.management.deploy.classdeployment.ClassDeploymentManager.access$000(ClassDeploymentManager.java:54)
at weblogic.management.deploy.classdeployment.ClassDeploymentManager$1.run(ClassDeploymentManager.java:226)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
at weblogic.management.deploy.classdeployment.ClassDeploymentManager.invokeClassDeployment(ClassDeploymentManager.java:258)
at weblogic.management.deploy.classdeployment.ClassDeploymentManager.runStartupsBeforeAppDeployments(ClassDeploymentManager.java:149)
at weblogic.management.deploy.classdeployment.ClassDeploymentService.start(ClassDeploymentService.java:21)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

SOLUTION:
Just edit the nodemanager.properties file that is inside the $WL_HOME/common/nodemanager.
Find the line with the content “StartScriptEnabled=false” and replace to “StartScriptEnabled=true”

If do you prefer, just run the command:

sed -i ‘s/StartScriptEnabled=false/StartScriptEnabled=true/g’ $WL_HOME/common/nodemanager/nodemanager.properties

Application | Middleware | GoldenGate, GoldenGate

GoldenGate 12.3

AYE! GoldenGate 12.3 is now released!

http://www.oracle.com/technetwork/middleware/goldengate/downloads/index.html

Capturar

What it brings?

– Support for Oracle Database 12.2.0.1
– Microservices Architecture
– Parallel Replicat
– Automatic Conflict-Detection-Resolution (CDR) without application changes
– Procedural Replication to enable simpler application migrations and upgrades
– Database Sharding
– Fast Start Capture

Let’s test this new features 🙂

Cheers!

Database, Errors and Bugs

Logs Exploding: ORA-00904: “UTL_RAW”.”CAST_FROM_NUMBER”: invalid identifier

 ,

Hello all!
Last week I suddenly had several database connections being denied due “ORA-09925: Unable to create audit trail file”. This just a few hours a maintenance in this Database.

When investigating, realized there was TO MUCH traces like “$DBNAME_q00%.trc using several GBs.
As you know, those traces are from QMON (Queue Monitor) Background Process, usually related to Oracle Streams Advanced Queueing.
Oh! Also noticed that database was started with srvctl, as per usually recommended.

After some research I found MOS Note: ORA-00904 Reported by QMON When Starting Database With SRVCTL (Doc ID 1950142.1).
This situation seems to be a match to bug Bug 18680601/19995869. Regarding Doc:
– As solution, Oracle recommends apply merge of patches 19995869 and patch 18628216.
– As immediate action, the workaround is shutdown database and start with sqlplus instead of srvctl.

Check below one of my trace files to match to your situation.

Hope it helps.

Cheers!

Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
ORACLE_HOME = /app/oracle/product/11.2.0.4
System name:    Linux
Node name:      racgrepora1
Release:        2.6.32-642.15.1.el6.x86_64
Version:        #1 SMP Mon Feb 20 02:26:38 EST 2017
Machine:        x86_64
Instance name: GREPORADB
Redo thread mounted by this instance: 1
Oracle process number: 77
Unix process pid: 36422, image: oracle@racgrepora1 (Q001)


*** 2017-08-01 05:05:51.352
*** SESSION ID:(646.64779) 2017-08-01 05:05:51.352
*** CLIENT ID:() 2017-08-01 05:05:51.352
*** SERVICE NAME:(SYS$BACKGROUND) 2017-08-01 05:05:51.352
*** MODULE NAME:(Streams) 2017-08-01 05:05:51.352
*** ACTION NAME:(QMON Slave) 2017-08-01 05:05:51.352

KSV 12801 error in slave process

*** 2017-08-01 05:05:51.352
ORA-12801: error signaled in parallel query server PZ99, instance racwt22:JTS012 (2)
ORA-00904: "UTL_RAW"."CAST_FROM_NUMBER": invalid identifier
OPIRIP: Uncaught error 447. Error stack:
ORA-00447: fatal error in background process
ORA-12801: error signaled in parallel query server PZ99, instance racwt22:JTS012 (2)
ORA-00904: "UTL_RAW"."CAST_FROM_NUMBER": invalid identifier
Database, Enterprise Manager

OEM Alarm – %MB of Audit Trail files (sizeOfOSAuditFiles:FILE_SIZE)

 , , ,

Hello All,
After upgrading a OEM to 13c, I started to receive notifications for event “sizeOfOSAuditFiles:FILE_SIZE“.

This is a new event implemented on OEM DB Plugin 12.1.0.7.0 under “Operating System Audit Records” metric group. Upgrading DB Plugin was part of OEM Upgrade change once we had some old versioned.

This event is only a notification related to file size for space management ends. The default thresholds are 10MB (warning) and 20MB (critical), which in most of times it’s a pretty low value.
This is specifically related to location under parameter audit_file_dest if you want to check.

Between options to reduce the noise are disable this metric or increase thresholds accordingly, which was what I did.
At this moment, I just increased thresholds to 500MB/2048MB, which I consider good values for the environment.

Some reference about can be found at:
– Enterprise Manager Oracle Database Plug-in Metric Reference Manual (Plug-in Release 12.1.0.7) – Database Instance – Operating System Audit Records
– EM 12c, EM 13c: Troubleshooting Database Metrics in Enterprise Manager 12c and 13c Cloud Control (Doc ID 2032156.1)

Hope it helps,
Cheers!

Application | Middleware | GoldenGate, Weblogic

Weblogic very slow during startup

 , , ,

Why does my Weblogic Server takes a Long Time to Start?

On Linux or Solaris operating systems, this behavior is very common. Especially in new installations.
There is more information about this behavior in “Doc ID 1574979.1“.

The solution to this problem is very simple.
Just edit the java.security file that is inside the $$JAVA_HOME/jre/lib/security/
Find the line with the content “securerandom.source=file:/dev/urandom” and replace with “securerandom.source=file:/dev/./urandom”.

If do you prefer, just run the command:

sed -i ‘s/securerandom.source=file:\/dev\/urandom/securerandom.source=file:\/dev\/.\/urandom/g’ $JAVA_HOME/jre/lib/security/java.security

 

Enterprise Manager

OEM Alarms: TaskZombieException

 , , ,

Hello all,
Recently I started to have several alarms like this:

Internal error detected: java.lang.Throwable:oracle.sysman.gcagent.tmmain.execution.LongOpManager$ZombieDetection:1017.

Or

Internal error detected: oracle.sysman.gcagent.task.TaskZombieException:oracle.sysman.gcagent.task.TaskFutureImpl$WrappedTask:620.

In agent log the message:

# $AGENT_INST/sysman/log/gcagent.log:

2017-06-13 12:54:09,232 [355:GC.Executor.14 (oracle_database:DB_DB12:%DB%)] ERROR - oracle_database:DB_DB12:%DB% oracle.sysman.gcagent.task.TaskZombieException: task declared as a zombie
at oracle.sysman.gcagent.task.TaskFutureImpl$WrappedTask.accountedCall(TaskFutureImpl.java:620)
at oracle.sysman.gcagent.task.TaskFutureImpl$WrappedTask.call(TaskFutureImpl.java:643)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)

To avoid generating pages and new incidents, I changed parameter _zombieCreateIncident to false on agents in related servers, as per described in MOS EM12c: Incident constantly raised for Oracle.sysman.gcagent.task.TaskZombieException: task declared as a zombie (Doc ID 2116834.1) regarding MOS Bug 22674258 – Zombie processes created for DB metric collection /workaround do not work.
I also added some other parameters that should help, like increasing the wait, all mentioned in same MOS notes.

Like this, in $AGENT_INST/sysman/config/emd.properties:

#GrepOra magical fixes:
_zombieSuspensions=true
_canceledThreadWait=900
_zombieThreadPercentThreshold=0
_zombieCreateIncident=false

And it solved the issue. At leat, the noise stopped. 🙂

Hope it helps,
Cheers!

Community, Events

GUOB Tech Day 2017 Review!

 , , ,

Hi all!

This year I had again the privilege to be in GUOB 2017 as a Speaker. Check below the excellent folks there (all speakers):

guob_speakers

As this is a pretty quick post, I can only say thank you for all reception I had by organization and colleges. I’m not giving a detailed feedback on each session but I regret nothing on this year, all choices were excellent! 😀

# Sessions I attended:

– Upgrade to Oracle Database 12.2 – Live and Uncensored (Mike Dietrich)
– DevOps for DBAs (Bjoern Rost)
– Database Security com TRansparent Data Encryption (Adriano Bonacin)
– Moving your Oracle Databases to Oracle Cloud (Alex Zaballa)
РImplanta̤̣o de BI Cloud Service (Carina Mendes)
– Oracle Database Backup Service: Backup de Banco de Dados On-Premise para Cloud (Matheus Boesing)
– Extraindo o Máximo do Oracle Grid Infraestructure (Franky Weber Faust)
– How to Diagnose Ramdom Oracle Cloud Performance Incidents Using ASH (Craig Shallahamer)

An that’s it! See you next year!

guob_Galera

Enterprise Manager

OEM 12c+: Prevent OMS from Sending Old Notifications for Events

 , , , ,

Are you receiving old notifications from OEM? Like 2 or 3 days past, mostly already solved or after a blackout?
Yeah, this is annoying, specially when getting floods and floods of notifications.

Ok, so here go a very good tip: You can set grace period for notifications! 🙂
Easy easy, do this way:

cd /bin
emctl set property -name oracle.sysman.core.notification.grace_period -value [provide value in minutes]

The oracle.sysman.core.notification.grace_period OMS parameter has been introduced in 12c and allows the user to configure the grace period within which the notification should be sent. The value is set in minutes.

For example:

emctl set property -name oracle.sysman.core.notification.grace_period -value 1440

With this, OMS sends only those notifications which have been raised in the past 1440 mins (last 24 hours) and ignores all the notifications for events / incidents created prior to this time period.

After this, you’ll need to start OMS:

emctl start oms

The oracle.sysman.core.notification.grace_period parameter applies to all the Notification methods, but if the requirement is to specify the grace period for a particular notification method only, you can use the below parameters accordingly:

oracle.sysman.core.notification.grace_period_connector: For Connectors
oracle.sysman.core.notification.grace_period_email: For email notifications
oracle.sysman.core.notification.grace_period_oscmd: For OS Command notifications
oracle.sysman.core.notification.grace_period_plsql: For PLSQL notifications
oracle.sysman.core.notification.grace_period_snmp: For SNMP Trap notifications
oracle.sysman.core.notification.grace_period_ticket: For ticketing tools

This is weel described as per MOS: 12c Cloud control: How to Prevent OMS from Sending out Old Notifications for Events / Incidents Occurred in the Past? (Doc ID 1605351.1)

Cheers!