The cluster upgrade state is [ROLLING PATCH] with correct Patch Level in all nodes

Hi all,
When performing a spot health check in a client environment, got this:

[oracle@dbserver1 ~]$ crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [12.1.0.2.0]
[oracle@dbserver1 ~]$ crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [12.1.0.2.0]. The cluster upgrade state is [ROLLING PATCH]. The cluster active patch level is [26717470].

By checking over the applied patched on the 3 nodes, it’s matching:

+ASM1@dbserver1 > kfod op=patches
List of Patches
===============
20243804
20415006
20594149
20788771
20950328
21125181
21359749
21436941
21527488
21694919
21949015
22806133
23144544
24007012
24340679
24732088
24846605
25397136
25869760
26392164
26392192
26609798
26717470

+ASM2@dbserver2 > kfod op=patches
List of Patches
===============
20243804
20415006
20594149
20788771
20950328
21125181
21359749
21436941
21527488
21694919
21949015
22806133
23144544
24007012
24340679
24732088
24846605
25397136
25869760
26392164
26392192
26609798
26717470

+ASM3@dbserver3 > kfod op=patches
List of Patches
===============
20243804
20415006
20594149
20788771
20950328
21125181
21359749
21436941
21527488
21694919
21949015
22806133
23144544
24007012
24340679
24732088
24846605
25397136
25869760
26392164
26392192
26609798
26717470

So, it’s most likely some patch completed wrongly. Here is the quick fix:

$GI_HOME/bin/clscfg -patch
$GI_HOME/bin/crsctl stop rollingpatch

Once done, issue fixed!
I hope it helps!

Relying in Guaranteed Restore Points? Be careful!

Hi all,

Are you relying on Guaranteed Restore Points (GRP) as a fallback plan for your migration or upgrade strategy? Be careful!

When performing some non-Prod upgrade with the Autoupgrade tool,  after completing the upgrade, I wanted to roll it back and go through the process again,  This is what happened:

SQL> startup
ORA-29702: error occurred in Cluster Group Service operation

When looking for it found this blog post from Mike I missed the last year: https://mikedietrichde.com/2020/11/13/ora-29702-and-your-instance-does-not-startup-in-the-cluster-anymore/

This means my database is not starting anymore! Oh man, glad that I’m in the testing phase!

This caused by of Bug 31561819 – Incompatible maxmembers at CRSD Level Causing Database Instance Not Able to Start.

As per Mike’s post, “you don’t need to even restore or flashback a database to hit this error. A simple instance in NOMOUNT state leads to the same error. Without even any datafile.”

The bug is fixed on:

  • 19.9.0.0.201020 (Oct 2020) OCW RU
  • 18.12.0.0.201020 (Oct 2020) OCW RU
  • 12.2.0.1.201020 (Oct 2020) OCW RU

As being, you should include this patch BEFORE starting any move! Do it right away if you are on these versions!

Also, be aware of the latest change regarding Restore Points propagation on 19c, as per MOS Automatic Propagate Restore Points from Primary to Standby site in 19C (Doc ID 2463082.1)

In my case, the usage is exactly for a 12.1->19c upgrade. So, the fix is not even available (no Extended Support in place). As being, we had to think on alternate fallback plans, like a physical standby. But this is a topic for another post.

So for YOU:

  • Apply this patch if you can!
  • If not, be very careful on the fallback plans and as usual: Test, Test and Test!

See you next post!

Opatchauto Failing on “CheckActiveFilesAndExecutables” during Prerequisite Check

Hi all,
So, very recently when applying the 2021 January CPU in a client environment, the following happened:

[root@dbserver01 32226239]# $ORACLE_HOME/OPatch/opatchauto apply

OPatchauto session is initiated at Sun Mar 14 03:00:06 2021

System initialization log file is /u01/app/oracle/product/19c/grid/cfgtoollogs/opatchautodb/systemconfig2021-03-14_03-00-08AM.log.

Session log file is /u01/app/oracle/product/19c/grid/cfgtoollogs/opatchauto/opatchauto2021-03-14_03-00-13AM.log
The id for this session is 1J89

Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19c/db
Patch applicability verified successfully on home /u01/app/oracle/product/19c/db


Executing patch validation checks on home /u01/app/oracle/product/19c/db
Patch validation checks successfully completed on home /u01/app/oracle/product/19c/db


Verifying SQL patch applicability on home /u01/app/oracle/product/19c/db
SQL patch applicability verified successfully on home /u01/app/oracle/product/19c/db


Executing OPatch prereq operations to verify patch applicability on home /u01/app/oracle/product/19c/grid
Patch applicability verified successfully on home /u01/app/oracle/product/19c/grid


Executing patch validation checks on home /u01/app/oracle/product/19c/grid
Patch validation checks successfully completed on home /u01/app/oracle/product/19c/grid


Preparing to bring down database service on home /u01/app/oracle/product/19c/db
Successfully prepared home /u01/app/oracle/product/19c/db to bring down database service


Bringing down database service on home /u01/app/oracle/product/19c/db
Following database has been stopped and will be restarted later during the session: er1pprd,obiee
Database service successfully brought down on home /u01/app/oracle/product/19c/db


Performing prepatch operations on CRS - bringing down CRS service on home /u01/app/oracle/product/19c/grid
Prepatch operation log file location: /u01/app/oracle/product/crsdata/dbserver01/crsconfig/hapatch_2021-03-14_03-06-15AM.log
CRS service brought down successfully on home /u01/app/oracle/product/19c/grid


Start applying binary patch on home /u01/app/oracle/product/19c/db
Failed while applying binary patches on home /u01/app/oracle/product/19c/db

Execution of [OPatchAutoBinaryAction] patch action failed, check log for more details. Failures:
Patch Target : dbserver01->/u01/app/oracle/product/19c/db Type[sidb]
Details: [
---------------------------Patching Failed---------------------------------
Command execution failed during patching in home: /u01/app/oracle/product/19c/db, host: dbserver01.
Command failed: /u01/app/oracle/product/19c/db/OPatch/opatchauto apply /ora02/soft/jan21cpu/32126842/32226239 -oh /u01/app/oracle/product/19c/db -target_type oracle_database -binary -invPtrLoc /u01/app/oracle/product/19c/grid/oraInst.loc -jre /u01/app/oracle/product/19c/grid/OPatch/jre -persistresult /u01/app/oracle/product/19c/db/opatchautocfg/db/sessioninfo/sessionresult_dbserver01_sidb_2.ser -analyzedresult /u01/app/oracle/product/19c/db/opatchautocfg/db/sessioninfo/sessionresult_analyze_dbserver01_sidb_2.ser
Command failure output:
==Following patches FAILED in apply:

Patch: /ora02/soft/jan21cpu/32126842/32226239/32218454
Log: /u01/app/oracle/product/19c/db/cfgtoollogs/opatchauto/core/opatch/opatch2021-03-14_03-17-58AM_1.log
Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: Prerequisite check "CheckActiveFilesAndExecutables" failed.

After fixing the cause of failure Run opatchauto resume

]
OPATCHAUTO-68061: The orchestration engine failed.
OPATCHAUTO-68061: The orchestration engine failed with return code 1
OPATCHAUTO-68061: Check the log for more details.
OPatchAuto failed.

OPatchauto session completed at Sun Mar 14 03:19:25 2021
Time taken to complete the session 19 minutes, 19 seconds

opatchauto failed with error code 42

OK, going by parts, let's see what we have on the refered log:

[Mar 14, 2021 3:33:32 AM] [INFO] Start fuser command /sbin/fuser /u01/app/oracle/product/19c/grid/bin/expdp at Sat Mar 14 03:33:32 PDT 2021
[Mar 14, 2021 3:33:32 AM] [INFO] Finish fuser command /sbin/fuser /u01/app/oracle/product/19c/grid/bin/expdp at Sat Mar 14 03:33:32 PDT 2021
[Mar 14, 2021 3:33:32 AM] [INFO] Following active executables are not used by opatch process :


Following active executables are used by opatch process :
/u01/app/oracle/product/19c/grid/lib/libclntsh.so.19.1
[Mar 14, 2021 3:33:32 AM] [INFO] Prerequisite check "CheckActiveFilesAndExecutables" failed.
The details are:


Following active executables are not used by opatch process :


Following active executables are used by opatch process :
/u01/app/oracle/product/19c/grid/lib/libclntsh.so.19.1
[Mar 14, 2021 3:33:33 AM] [INFO] UtilSession failed: Prerequisite check "CheckActiveFilesAndExecutables" failed.
[Mar 14, 2021 3:33:33 AM] [SEVERE] OUI-67073:UtilSession failed: Prerequisite check "CheckActiveFilesAndExecutables" failed.
[Mar 14, 2021 3:33:33 AM] [INFO] Finishing UtilSession at Sat Mar 14 03:33:33 PDT 2021
[Mar 14, 2021 3:33:33 AM] [INFO] Log file location: /u01/app/oracle/product/19c/grid/cfgtoollogs/opatchauto/core/opatch/oapatch_2021-03-14_03-06-15AM.log

This is an interesting situation.

After some validations making sure no service is online, the path is writable, oracle and root have the required privilege and access, I found some relevant Oracle notes:

  • 19c Installation Fails with error “libclntsh.so: file format not recognized; treating as linker script” (Doc ID 2631283.1): Pointing to file corruption
  • While Applying a Weblogic Patch, opatch Fails with “Prerequisite check “CheckActiveFilesAndExecutables” failed” Error (Doc ID 2705809.1): Not a DB note and pointing to other processes using the files.
  • Opatch failure due to “CheckActiveFilesAndExecutables” as Remote registry service holding files (Doc ID 2462952.1): Remote registry holding the binaries.
  • Prerequisite Check “Checkactivefilesandexecutables” Failed (Doc ID 1281644.1): Patch requisite miss on 10g
  • Failed to apply PSU due to CheckActiveFilesAndExecutables check failure (Doc ID 2506432.1): SQLPlus holding the binaries.
  • [OCI]: Database System Patching Failed With Error “DCS-10001:Internal Error Encountered: Failure : Failed To Apply” And Opatch Log Shows “Prerequisite check “CheckActiveFilesAndExecutables” failed” (Doc ID 2687607.1): My case is not an OCI and not in RAC.

So, no matches at all.

However, this last note gave me the hints I needed. From Doc ID 2687607.1, for RAC environments:

/u01/app/19.0.0.0/grid/crs/install/rootcrs.sh -unlock
/u01/app/19.0.0.0/grid/crs/install/rootcrs.sh -init
/u01/app/19.0.0.0/grid/crs/install/rootcrs.sh -prepatch
/u01/app/19.0.0.0/grid/crs/install/rootcrs.sh -postpatch

So, in my case, a Standalone On-Premise Database (and GI):

/ora01/app/oracle/product/19c/grid/crs/install/roothas.sh -unlock
/ora01/app/oracle/product/19c/grid/crs/install/roothas.sh -init
/ora01/app/oracle/product/19c/grid/crs/install/roothas.sh -prepatch
[ Apply the patch! ]
/ora01/app/oracle/product/19c/grid/crs/install/roothas.sh -postpatch

Check the output:

[root@dbserver01 jan21cpu]# /u01/app/oracle/product/19c/grid/crs/install/roothas.sh -unlock
Using configuration parameter file: /u01/app/oracle/product/19c/grid/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/oracle/product/crsdata/dbserver01/crsconfig/haunlock__2021-03-14_04-00-35AM.log
2021/03/14 04:01:01 CLSRSC-347: Successfully unlock /u01/app/oracle/product/19c/grid
[root@dbserver01 jan21cpu]# /u01/app/oracle/product/19c/grid/crs/install/roothas.sh -init
Using configuration parameter file: /u01/app/oracle/product/19c/grid/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/oracle/product/crsdata/dbserver01/crsconfig/roothas_2021-03-14_04-01-09AM.log
[root@dbserver01 jan21cpu]# /u01/app/oracle/product/19c/grid/crs/install/roothas.sh -prepatch
Using configuration parameter file: /u01/app/oracle/product/19c/grid/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/oracle/product/crsdata/dbserver01/crsconfig/hapatch_2021-03-14_04-01-16AM.log
2021/03/14 04:01:27 CLSRSC-347: Successfully unlock /u01/app/oracle/product/19c/grid
2021/03/14 04:01:27 CLSRSC-671: Pre-patch steps for patching GI home successfully completed.

And now resuming the Opatchauto:

[root@dbserver01 jan21cpu]# cd 32126842/32226239/
[root@dbserver01 32226239]# $ORACLE_HOME/OPatch/opatchauto resume

OPatchauto session is initiated at Sun Mar 14 04:02:07 2021
Session log file is /u01/app/oracle/product/19c/grid/cfgtoollogs/opatchauto/opatchauto2021-03-14_04-02-10AM.log
Resuming existing session with id 1J89

Start applying binary patch on home /u01/app/oracle/product/19c/db
Binary patch applied successfully on home /u01/app/oracle/product/19c/db


Start applying binary patch on home /u01/app/oracle/product/19c/grid

Binary patch applied successfully on home /u01/app/oracle/product/19c/grid


Performing postpatch operations on CRS - starting CRS service on home /u01/app/oracle/product/19c/grid
Postpatch operation log file location: /u01/app/oracle/product/crsdata/dbserver01/crsconfig/hapatch_2021-03-14_04-27-58AM.log
CRS service started successfully on home /u01/app/oracle/product/19c/grid


Preparing home /u01/app/oracle/product/19c/db after database service restarted
No step execution required.........


Trying to apply SQL patch on home /u01/app/oracle/product/19c/db
SQL patch applied successfully on home /u01/app/oracle/product/19c/db

OPatchAuto successful.

--------------------------------Summary--------------------------------

Patching is completed successfully. Please find the summary as follows:

Host:dbserver01
SIDB Home:/u01/app/oracle/product/19c/db
Version:19.0.0.0.0
Summary:

==Following patches were SKIPPED:

Patch: /ora02/soft/jan21cpu/32126842/32226239/32218663
Reason: This patch is not applicable to this specified target type - "oracle_database"

Patch: /ora02/soft/jan21cpu/32126842/32226239/29340594
Reason: This patch is not applicable to this specified target type - "oracle_database"

Patch: /ora02/soft/jan21cpu/32126842/32226239/32240590
Reason: This patch is not applicable to this specified target type - "oracle_database"


==Following patches were SUCCESSFULLY applied:

Patch: /ora02/soft/jan21cpu/32126842/32226239/32218454
Log: /u01/app/oracle/product/19c/db/cfgtoollogs/opatchauto/core/opatch/opatch2021-03-14_04-02-36AM_1.log

Patch: /ora02/soft/jan21cpu/32126842/32226239/32222571
Log: /u01/app/oracle/product/19c/db/cfgtoollogs/opatchauto/core/opatch/opatch2021-03-14_04-02-36AM_1.log


Host:dbserver01
SIHA Home:/u01/app/oracle/product/19c/grid
Version:19.0.0.0.0
Summary:

==Following patches were SUCCESSFULLY applied:

Patch: /ora02/soft/jan21cpu/32126842/32226239/29340594
Log: /u01/app/oracle/product/19c/grid/cfgtoollogs/opatchauto/core/opatch/opatch2021-03-14_04-11-34AM_1.log

Patch: /ora02/soft/jan21cpu/32126842/32226239/32218454
Log: /u01/app/oracle/product/19c/grid/cfgtoollogs/opatchauto/core/opatch/opatch2021-03-14_04-11-34AM_1.log

Patch: /ora02/soft/jan21cpu/32126842/32226239/32218663
Log: /u01/app/oracle/product/19c/grid/cfgtoollogs/opatchauto/core/opatch/opatch2021-03-14_04-11-34AM_1.log

Patch: /ora02/soft/jan21cpu/32126842/32226239/32222571
Log: /u01/app/oracle/product/19c/grid/cfgtoollogs/opatchauto/core/opatch/opatch2021-03-14_04-11-34AM_1.log

Patch: /ora02/soft/jan21cpu/32126842/32226239/32240590
Log: /u01/app/oracle/product/19c/grid/cfgtoollogs/opatchauto/core/opatch/opatch2021-03-14_04-11-34AM_1.log

OPatchauto session completed at Sun Mar 14 04:31:48 2021
Time taken to complete the session 29 minutes, 43 seconds

And here is the relevant point: This has been happening to me on several environments and servers across the recent weeks. Always for 2021 January CPU.
My guess is that this might have something to do with this CPU binaries set or, most likely, with the latest OPatch version:

[oracle@dbserver01 ~]$ $ORACLE_HOME/OPatch/opatch version
OPatch Version: 12.2.0.1.24

I hope it helps you as well!

OPatch Auto Failing to Apply Datapatch for a Removed Database

Hi all,

So, just a few days ago, during a client support, crossed the following case.

A few database creations initially failed with DBCA due to other issues, but it seems the DBCA didn’t cleread all creation steps after the failure and theoretical rollback.

As a consequence, whenever running DBCA with GUI client was seeing an old database. When trying to remove it with DBCA, the removal fail as the database can’t be brought up (creation has failed, remember?). Ok, we can leave with it, right?

Yes until we reached the point where Opatchauto failed with the following:

Verifying SQL patch applicability on home /u01/app/oracle/product/19c/db

"/bin/sh -c 'cd /u01/app/oracle/product/19c/db; ORACLE_HOME=/u01/app/oracle/product/19c/db ORACLE_SID=DB1 /u01/app/oracle/product/19c/db/OPatch/datapatch -prereq -verbose'" command failed with errors. Please refer to logs for more details. SQL changes, if any, can be analyzed by manually retrying the same command.

The reason? See the complete log about the failing step:

Executing command as oracle: 
 /bin/sh -c 'cd /u01/app/oracle/product/19c/db;ORACLE_HOME=/u01/app/oracle/product/19c/db ORACLE_SID=DB1 /u01/app/oracle/product/19c/db/OPatch/datapatch -verbose'
2020-09-02 16:26:56,362 INFO  [438] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - COMMAND Looks like this: /bin/sh -c 'cd /u01/app/oracle/product/19c/db;ORACLE_HOME=/u01/app/oracle/product/19c/db ORACLE_SID=DB1 /u01/app/oracle/product/19c/db/OPatch/datapatch -verbose'
2020-09-02 16:26:57,662 INFO  [438] com.oracle.glcm.patch.auto.db.product.executor.GISystemCall - Is retry required=false
2020-09-02 16:26:57,662 INFO  [438] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - status: 1
2020-09-02 16:26:57,662 INFO  [438] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - 
COMMAND EXECUTION FAILURE :
SQL Patching tool version 19.8.0.0.0 Production on Wed Sep  2 16:26:57 2020
Copyright (c) 2012, 2020, Oracle.  All rights reserved.

Log file for this invocation: /u01/app/oracle/base/cfgtoollogs/sqlpatch/sqlpatch_25218_2020_09_02_16_26_57/sqlpatch_invocation.log

Connecting to database...
Error: prereq checks failed!
Database connect failed with: ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Additional information: 4376
Additional information: 1275019259 (DBD ERROR: OCISessionBegin)

Please refer to MOS Note 1609718.1 and/or the invocation log
/u01/app/oracle/base/cfgtoollogs/sqlpatch/sqlpatch_25218_2020_09_02_16_26_57/sqlpatch_invocation.log
for information on how to resolve the above errors.

SQL Patching tool complete on Wed Sep  2 16:26:57 2020

Clearly, the database is still in place.
As per MOS (ORA-01078 Can Not Delete Database From Dbca on Linux (Doc ID 1559634.1)) -> See the /etc/oratab!

Thing is, the doesn’t have the DB1 line. Also, all related files, logs directory structure, passwd, init, etc… all wiped out. What else?

Here is goes what seems Oracle forgot to tell:

[oracle@PRODB01 dbca]$ srvctl status database -d DB1
Database is not running.
[oracle@DMSDB1PA dbca]$ srvctl config database -d DB1
Database unique name: DB1
Database name: DB1
Oracle home: /u01/app/oracle/product/19c/db
Oracle user: oracle
Spfile: 
Password file: 
Domain: 
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Disk Groups: DATA
Services: 
OSDBA group: oinstall
OSOPER group: 
Database instance: DB1
Ohhh, that took me a while to realize, this was a Standalone server. Once understood, the fix is straight forward:
[oracle@PRODB01 dbca]$ srvctl remove database -d DB1
Remove the database dms? (y/[n]) y
[oracle@PRODB01 dbca]$

I hope this can buy you some minutes of MOS, in case you are googling it first.
Or save you in case you gave up MOS already.

Cheers!

OEM 13C Patching Agent: [ERROR- Failed to Update Target Type Metadata]

While applying Patch to OEM13c, specifically to OMS Agent, I got this error when trying to start it back:

Collection Status                            : Collections enabled
Heartbeat Status       : OMS responded illegally [ERROR- Failed to Update Target Type Metadata]
Last attempted heartbeat to OMS              : 2018-04-17 12:12:51
Last successful heartbeat to OMS             : (none)
Next scheduled heartbeat to OMS              : 2018-04-17 12:13:21
---------------------------------------------------------------
Agent is Running and Ready
[oracle@oem13c oms]$ ./emctl upload agent
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD upload error:full upload has failed: uploadXMLFiles skipped :: OMS version not checked yet. If this issue persists check trace files for ping to OMS related errors. (OMS_DOWN)
[oracle@oem13c oms]$ ./emctl pingOMS
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD pingOMS error: OMS sent an invalid response: “ERROR- Failed to Update Target Type Metadata”

Nice hãn?
I Found to MOS EM 13c Agent : pingOMS error: OMS sent an invalid response: “ERROR- Failed to Update Target Type Metadata” (Doc ID 2318564.1) saying:

“This particular issue is caused when any Agent Plugin is upgraded to a higher level than the OMS plugin.”

The solution according to MOS Doc is to rollback the Agent Plugins ahead to the OMS version. Checking it:
More“OEM 13C Patching Agent: [ERROR- Failed to Update Target Type Metadata]”

ORA-00600: [qkswcWithQbcRefdByMain4]

Hello all,
This days I found this in a client’s 12c Database when trying to create a Materialized View:

ORA-00600: internal error code, arguments: [qkswcWithQbcRefdByMain4]

A perfect match to MOS ORA-00600 [qkswcWithQbcRefdByMain4] when Create MV “WITH” clause (Doc ID 2232872.1).

The root cause is documented on BUG 22867413 – ORA-600 CALLING DBMS_ADVISOR.TUNE_MVIEW.
The given solution is to apply Patch 22867413.

After applying patch, issue solved. 🙂

More“ORA-00600: [qkswcWithQbcRefdByMain4]”

Online Data Patch Apply with multiple Databases on same Oracle Home: OPatch failed with error code 26

Hi all,
Tricky question, right? It’s easier than you think…
Actually, we don’t commonly think on those situations in first place, but it’s pretty common, specialy if considering server consolidation situations.

The trick is to use clause util enableOnlinePatch insted of apply after first database applying.
In this example I’m applying on-off patch 14084247 in online mode. Check:

# First Database:

[oracle@PRODSERVER 14084247]$ opatch apply online -connectString ORA11:sys::
Oracle Interim Patch Installer version 11.2.0.3.4
Copyright (c) 2012, Oracle Corporation.  All rights reserved.


Oracle Home       : /opt/oracle/app/product/11.2.0/db_1
Central Inventory : /opt/oracle/oraInventory
   from           : /opt/oracle/app/product/11.2.0/db_1/oraInst.loc
OPatch version    : 11.2.0.3.4
OUI version       : 11.2.0.4.0
Log file location : /opt/oracle/app/product/11.2.0/db_1/cfgtoollogs/opatch/14084247_Apr_03_2017_14_17_24/apply2017-04-03_14-17-24PM_1.log


The patch should be applied/rolled back in '-all_nodes' mode only.
Converting the RAC mode to '-all_nodes' mode.
Applying interim patch '14084247' to OH '/opt/oracle/app/product/11.2.0/db_1'
Verifying environment and performing prerequisite checks...
All checks passed.
Backing up files...

Patching component oracle.rdbms, 11.2.0.4.0...
Installing and enabling the online patch 'bug14084247.pch', on database 'ORA11'.


Verifying the update...
Patch 14084247 successfully applied
Log file location: /opt/oracle/app/product/11.2.0/db_1/cfgtoollogs/opatch/14084247_Apr_03_2017_14_17_24/apply2017-04-03_14-17-24PM_1.log

OPatch succeeded.

All good, right?
Let’s see applying to second database with same command:

Second Database:

[oracle@PRODSERVER 14084247]$ opatch apply online -connectString OTHERORA11:sys::
Oracle Interim Patch Installer version 11.2.0.3.4
Copyright (c) 2012, Oracle Corporation.  All rights reserved.


Oracle Home       : /opt/oracle/app/product/11.2.0/db_1
Central Inventory : /opt/oracle/oraInventory
   from           : /opt/oracle/app/product/11.2.0/db_1/oraInst.loc
OPatch version    : 11.2.0.3.4
OUI version       : 11.2.0.4.0
Log file location : /opt/oracle/app/product/11.2.0/db_1/cfgtoollogs/opatch/14084247_Apr_03_2017_14_17_45/apply2017-04-03_14-17-45PM_1.log


The patch should be applied/rolled back in '-all_nodes' mode only.
Converting the RAC mode to '-all_nodes' mode.
Applying interim patch '14084247' to OH '/opt/oracle/app/product/11.2.0/db_1'
Verifying environment and performing prerequisite checks...
Log file location: /opt/oracle/app/product/11.2.0/db_1/cfgtoollogs/opatch/14084247_Apr_03_2017_14_17_45/apply2017-04-03_14-17-45PM_1.log

Recommended actions: Please use 'opatch util applySql' for sql related patches or 'opatch util enableOnlinePatch' for online patches to add sids to already installed patch(es).

OPatch failed with error code 26

Beeep!
So, simply use clause util enableonlinepatch as per below.

Second Database (right way):

[oracle@PRODSERVER 14084247]$ opatch util enableonlinepatch -connectString OTHERORA11:sys:: -id 14084247
Oracle Interim Patch Installer version 11.2.0.3.4
Copyright (c) 2012, Oracle Corporation. All rights reserved.

Oracle Home : /opt/oracle/app/product/11.2.0/db_1
Central Inventory : /opt/oracle/oraInventory
from : /opt/oracle/app/product/11.2.0/db_1/oraInst.loc
OPatch version : 11.2.0.3.4
OUI version : 11.2.0.4.0
Log file location : /opt/oracle/app/product/11.2.0/db_1/cfgtoollogs/opatch/opatch2017-04-03_14-20-53PM_1.log

Invoking utility "enableonlinepatch"
Installing and enabling the online patch 'bug14084247.pch', on database 'OTHERORA11'.

OPatch succeeded.
[oracle@PRODSERVER 14084247]$

Ok, that’s it for today.

See you next week!

PSUs for Databases 10.2.0.4 and above

Hi all!
I’m researching about and decided to share the list of all Oracle Patch Set Updates (PSU) for Databases 10.2.0.4 and above, until now.
It’s always good to know the last PSU for every version to better fit on our patching policy.

Note: To understand database version numbers: https://docs.oracle.com/cd/B28359_01/server.111/b28310/dba004.htm

I expect it be useful for you too!

Patches for 12.1.0.2
Patch 21359755 – DATABASE PATCH SET UPDATE 12.1.0.2.5
Patch 21523234 – GRID INFRASTRUCTURE PATCH SET UPDATE 12.1.0.2.5 (OCT2015)
Patch 20831110 – DATABASE PATCH SET UPDATE 12.1.0.2.4 (INCLUDES CPUJUL2015)
Patch 20996835 – GRID INFRASTRUCTURE PATCH SET UPDATE 12.1.0.2.4 (JUL2015)
Patch 20299023 – DATABASE PATCH SET UPDATE 12.1.0.2.3 (APR2015)
Patch 20485724 – GRID INFRASTRUCTURE PATCH SET UPDATE 12.1.0.2.3 (APR2015)
Patch 19769480 – DATABASE PATCH SET UPDATE 12.1.0.2.2 (JAN2015)
Patch 19954978 – GRID INFRASTRUCTURE PATCH SET UPDATE 12.1.0.2.2 (JAN2015)
Patch 19303936 – DATABASE PATCH SET UPDATE 12.1.0.2.1 (OCT2014)
Patch 19392646 – GRID INFRASTRUCTURE PATCH SET UPDATE 12.1.0.2.1 (OCT2014)

Patches for 12.1.0.1
Patch 21352619 – DATABASE PATCH SET UPDATE 12.1.0.1.9
Patch 21551666 – GRID INFRASTRUCTURE PSU 12.1.0.1.9 (OCT2015 – INCLUDES DB PSU 12.1.0.1.9)
Patch 20831107 – DATABASE PATCH SET UPDATE 12.1.0.1.8 (INCLUDES CPUJUL2015)
Patch 20996901 – GRID INFRASTRUCTURE PATCH SET UPDATE 12.1.0.1.8 (JUL2015)
Patch 20299016 – DATABASE PATCH SET UPDATE 12.1.0.1.7
Patch 20485762 – GRID INFRASTRUCTURE PATCH SET UPDATE 12.1.0.1.7 (APR2015)
Patch 19769486 – DATABASE PATCH SET UPDATE 12.1.0.1.6
Patch 19971324 – GRID INFRASTRUCTURE PATCH SET UPDATE 12.1.0.1.6 (JAN2015)
Patch 19121550 – DATABASE PATCH SET UPDATE 12.1.0.1.5
Patch 19392372 – GRID INFRASTRUCTURE PATCH SET UPDATE 12.1.0.1.5 (OCT2014)
Patch 18522516 – DATABASE PATCH SET UPDATE 12.1.0.1.4
Patch 18705901 – GRID INFRASTRUCTURE SYSTEM PATCH 12.1.0.1.4 (Exadata)
Patch 18705972 – GRID INFRASTRUCTURE SYSTEM PATCH 12.1.0.1.4
Patch 18031528 – DATABASE PATCH SET UPDATE 12.1.0.1.3
Patch 18139660 – GRID INFRASTRUCTURE SYSTEM PATCH 12.1.0.1.3 (Exadata)
Patch 18413105 – GRID INFRASTRUCTURE SYSTEM PATCH 12.1.0.1.3
Patch 17552800 – DATABASE PATCH SET UPDATE 12.1.0.1.2
Patch 17735306 – GRID INFRASTRUCTURE SYSTEM PATCH 12.1.0.1.2
Patch 17027533 – DATABASE PATCH SET UPDATE 12.1.0.1.1
Patch 17272829 – GRID INFRASTRUCTURE PSU 12.1.0.1.1

Patches for 11.2.0.4
Patch 21352635 – DATABASE PATCH SET UPDATE 11.2.0.4.8 (INCLUDES CPUOCT2015)
Patch 21523375 – GRID INFRASTRUCTURE PATCH SET UPDATE 11.2.0.4.8 (OCT2015)
Patch 20760982 – DATABASE PATCH SET UPDATE 11.2.0.4.7 (INCLUDES CPUJUL2015)
Patch 20996923 – GRID INFRASTRUCTURE PATCH SET UPDATE 11.2.0.4.7 (JUL2015)
Patch 20299013 – DATABASE PATCH SET UPDATE 11.2.0.4.6 (INCLUDES CPUAPR2015)
Patch 20485808 – GRID INFRASTRUCTURE PATCH SET UPDATE 11.2.0.4.6 (APR2015)
Patch 19769489 – DATABASE PATCH SET UPDATE 11.2.0.4.5 (INCLUDES CPUJAN2015)
Patch 19955028 – GRID INFRASTRUCTURE PATCH SET UPDATE 11.2.0.4.5 (JAN2015)
Patch 19121551 – DATABASE PATCH SET UPDATE 11.2.0.4.4 (INCLUDES CPUOCT2014)
Patch 19380115 – GRID INFRASTRUCTURE SYSTEM PATCH 11.2.0.4.4
Patch 18522509 – DATABASE PATCH SET UPDATE 11.2.0.4.3 (INCLUDES CPUJUL2014)
Patch 18706472 – GRID INFRASTRUCTURE SYSTEM PATCH 11.2.0.4.3
Patch 18031668 – DATABASE PATCH SET UPDATE 11.2.0.4.2 (INCLUDES CPUAPR2014)
Patch 18139609 – GRID INFRASTRUCTURE SYSTEM PATCH 11.2.0.4.2
Patch 17478514 – DATABASE PATCH SET UPDATE 11.2.0.4.1 (INCLUDES CPUJAN2014)

Patches for 11.2.0.3
Patch 20760997 – DATABASE PATCH SET UPDATE 11.2.0.3.15 (INCLUDES CPUJUL2015)
Patch 20996944 – GRID INFRASTRUCTURE PATCH SET UPDATE 11.2.0.3.15 (JUL2015)
Patch 20299017 – DATABASE PATCH SET UPDATE 11.2.0.3.14 (INCLUDES CPUAPR2015)
Patch 20485830 – GRID INFRASTRUCTURE PATCH SET UPDATE 11.2.0.3.14 (APR2015)
Patch 19769496 – DATABASE PATCH SET UPDATE 11.2.0.3.13 (INCLUDES CPUJAN2015)
Patch 19971343 – GRID INFRASTRUCTURE PSU 11.2.0.3.13 (JAN2015)
Patch 19121548 – DATABASE PATCH SET UPDATE 11.2.0.3.12 (INCLUDES CPUOCT2014)
Patch 19440385 – GRID INFRASTRUCTURE PSU 11.2.0.3.12 (INCLUDES DB PSU 11.2.0.3.12)
Patch 18522512 – DATABASE PATCH SET UPDATE 11.2.0.3.11 (INCLUDES CPUJUL2014)
Patch 18706488 – GRID INFRASTRUCTURE PSU 11.2.0.3.11 (INCLUDES DB PSU 11.2.0.3.11)
Patch 18031683 – DATABASE PATCH SET UPDATE 11.2.0.3.10
Patch 18139678 – GRID INFRASTRUCTURE PSU 11.2.0.3.10 (INCLUDES DB PSU 11.2.0.3.10)
Patch 17540582 – DATABASE PATCH SET UPDATE 11.2.0.3.9 (INCLUDES CPUJAN2014)
Patch 17735354 – GRID INFRASTRUCTURE PSU 11.2.0.3.9 (INCLUDES DB PSU 11.2.0.3.9)
Patch 16902043 – DATABASE PATCH SET UPDATE 11.2.0.3.8 (INCLUDES CPUOCT2013)
Patch 17272731 – GRID INFRASTRUCTURE PSU 11.2.0.3.8 (INCLUDES DB PSU 11.2.0.3.8)
Patch 16619892 – DATABASE PATCH SET UPDATE 11.2.0.3.7 (INCLUDES CPUJUL2013)
Patch 16742216 – GRID INFRASTRUCTURE PSU 11.2.0.3.7 (INCLUDES DB PSU 11.2.0.3.7)
Patch 16056266 – DATABASE PATCH SET UPDATE 11.2.0.3.6 (INCLUDES CPUAPR2013)
Patch 16083653 – GRID INFRASTRUCTURE PSU 11.2.0.3.6 (INCLUDES DB PSU 11.2.0.3.6)
Patch 14727310 – DATABASE PATCH SET UPDATE 11.2.0.3.5 (INCLUDES CPUJAN2013)
Patch 14727347 – GRID INFRASTRUCTURE PSU 11.2.0.3.5 (INCLUDES DB PSU 11.2.0.3.5)
Patch 14275605 – DATABASE PATCH SET UPDATE 11.2.0.3.4 (INCLUDES CPUOCT2012)
Patch 14275572 – GRID INFRASTRUCTURE PSU 11.2.0.3.4 (INCLUDES DB PSU 11.2.0.3.4)
Patch 13923374 – DATABASE PATCH SET UPDATE 11.2.0.3.3 (INCLUDES CPU JUL2012)
Patch 13919095 – GRID INFRASTRUCTURE PSU 11.2.0.3.3 (INCLUDES DB PSU 11.2.0.3.3)
Patch 13696216 – DATABASE PATCH SET UPDATE 11.2.0.3.2 (INCLUDES CPU APR2012)
Patch 13696251 – GRID INFRASTRUCTURE PSU 11.2.0.3.2 (INCLUDES DB PSU 11.2.0.3.2)
Patch 13343438 – DATABASE PATCH SET UPDATE 11.2.0.3.1 (INCLUDES CPU JAN2012)

Patches for 11.2.0.2
Patch 17082367 – DATABASE PATCH SET UPDATE 11.2.0.2.12 (INCLUDES CPUOCT2013)
Patch 17272753 – GRID INFRASTRUCTURE PSU 11.2.0.2.12 (INCLUDES DB PSU 11.2.0.2.12)
Patch 16619893 – DATABASE PATCH SET UPDATE 11.2.0.2.11 (INCLUDES CPUJUL2013)
Patch 16742320 – GRID INFRASTRUCTURE PSU 11.2.0.2.11 (INCLUDES DB PSU 11.2.0.2.11)
Patch 16056267 – DATABASE PATCH SET UPDATE 11.2.0.2.10 (INCLUDES CPUAPR2013)
Patch 16166868 – GRID INFRASTRUCTURE PSU 11.2.0.2.10 (INCLUDES DB PSU 11.2.0.2.10)
Patch 14727315 – DATABASE PATCH SET UPDATE 11.2.0.2.9 (INCLUDES CPUJAN2013)
Patch 14841385 – GRID INFRASTRUCTURE PSU 11.2.0.2.9 (INCLUDES DB PSU 11.2.0.2.9)
Patch 14275621 – DATABASE PATCH SET UPDATE 11.2.0.2.8 (INCLUDES CPUOCT2012)
Patch 14390437 – GRID INFRASTRUCTURE PSU 11.2.0.2.8 (INCLUDES DB PSU 11.2.0.2.8)
Patch 13923804 – DATABASE PATCH SET UPDATE 11.2.0.2.7 (INCLUDES CPU JUL2012)
Patch 14192201 – GRID INFRASTRUCTURE PSU 11.2.0.2.7 (INCLUDES DB PSU 11.2.0.2.7)
Patch 13696224 – DATABASE PATCH SET UPDATE 11.2.0.2.6 (INCLUDES CPU APR2012)
Patch 13696242 – GRID INFRASTRUCTURE PSU 11.2.0.2.6 (INCLUDES DB PSU 11.2.0.2.6)
Patch 13343424 – DATABASE PATCH SET UPDATE 11.2.0.2.5 (INCLUDES CPU JAN2012)
Patch 12827726 – DATABASE PSU 11.2.0.2.4 (INCLUDES CPUOCT2011)
Patch 12419331 – DATABASE PSU 11.2.0.2.3 (INCLUDES CPUJUL2011)
Patch 11724916 – DATABASE PSU 11.2.0.2.2 (INCLUDES CPUAPR2011)
Patch 10248523 – DATABASE PSU 11.2.0.2.1 (Patch)

Patches for 11.2.0.1
Patch 12419378 – DATABASE PSU 11.2.0.1.6 (INCLUDES CPUJUL2011)
Patch 11724930 – DATABASE PSU 11.2.0.1.5 (INCLUDES CPUAPR2011)
Patch 10248516 – DATABASE PSU 11.2.0.1.4 (INCLUDES CPUJAN2011)
Patch 9952216 – DATABASE PSU 11.2.0.1.3 (INCLUDES CPUOCT2010)
Patch 9655006 – GI PSU 11.2.0.1.2 (INCLUDES DATABASE PSU 11.2.0.1.2)
Patch 9654983 – DATABASE PSU 11.2.0.1.2 (INCLUDES CPUJUL2010)
Patch 9352237 – DATABASE PSU 11.2.0.1.1 (Patch)

Patches for 11.1.0.7
Patch 20761024 – DATABASE PATCH SET UPDATE 11.1.0.7.24 (INCLUDES CPUJUL2015)
Patch 20299012 – DATABASE PATCH SET UPDATE 11.1.0.7.23 (INCLUDES CPUAPR2015)
Patch 19769499 – DATABASE PATCH SET UPDATE 11.1.0.7.22 (INCLUDES CPUJAN2015)
Patch 19152553 – DATABASE PATCH SET UPDATE 11.1.0.7.21 (INCLUDES CPUOCT2014)
Patch 18522513 – DATABASE PATCH SET UPDATE 11.1.0.7.20 (INCLUDES CPUJUL2014)
Patch 18031726 – DATABASE PATCH SET UPDATE 11.1.0.7.19 (INCLUDES CPUAPR2014)
Patch 17465583 – DATABASE PATCH SET UPDATE 11.1.0.7.18 (INCLUDES CPUJAN2014)
Patch 17082366 – DATABASE PATCH SET UPDATE 11.1.0.7.17 (INCLUDES CPUOCT2013)
Patch 16619896 – DATABASE PATCH SET UPDATE 11.1.0.7.16 (INCLUDES CPUJUL2013)
Patch 16056268 – DATABASE PATCH SET UPDATE 11.1.0.7.15 (INCLUDES CPUAPR2013)
Patch 14739378 – DATABASE PATCH SET UPDATE 11.1.0.7.14 (INCLUDES CPUJAN2013)
Patch 14275623 – DATABASE PATCH SET UPDATE 11.1.0.7.13 (INCLUDES CPUOCT2012)
Patch 13923474 – DATABASE PATCH SET UPDATE 11.1.0.7.12 (INCLUDES CPU JUL2012)
Patch 13621679 – DATABASE PATCH SET UPDATE 11.1.0.7.11 (INCLUDES CPU APR2012)
Patch 13343461 – DATABASE PATCH SET UPDATE 11.1.0.7.10 (INCLUDES CPU JAN2012)
Patch 12827740 – DATABASE PSU 11.1.0.7.9 (INCLUDES CPUOCT2011)
Patch 12419384 – DATABASE PSU 11.1.0.7.8 (INCLUDES CPUJUL2011)
Patch 11724953 – TRACKING BUG FOR 11.1.0.7.7 CRS PSU
Patch 11724936 – DATABASE PSU 11.1.0.7.7 (INCLUDES CPUAPR2011)
Patch 10248531 – DATABASE PSU 11.1.0.7.6 (INCLUDES CPUJAN2011)
Patch 9952228 – DATABASE PSU 11.1.0.7.5 (INCLUDES CPUOCT2010)
Patch 9654987 – DATABASE PSU 11.1.0.7.4 (INCLUDES CPUJUL2010)
Patch 9352179 – DATABASE PSU 11.1.0.7.3 (INCLUDES CPUAPR2010)
Patch 9209238 – DATABASE PSU 11.1.0.7.2 (INCLUDES CPUJAN2010)
Patch 8833297 – DATABASE PSU 11.1.0.7.1 (INCLUDES CPUOCT2009)

Patches for 10.2.0.5
Patch 16619894 – DATABASE PATCH SET UPDATE 10.2.0.5.12 (INCLUDES CPUJUL2013)
Patch 16056270 – DATABASE PATCH SET UPDATE 10.2.0.5.11 (INCLUDES CPUAPR2013)
Patch 14727319 – DATABASE PATCH SET UPDATE 10.2.0.5.10 (INCLUDES CPUJAN2013)
Patch 14275629 – DATABASE PATCH SET UPDATE 10.2.0.5.9 (INCLUDES CPUOCT2012)
Patch 13923855 – DATABASE PATCH SET UPDATE 10.2.0.5.8 (INCLUDES CPU JUL2012)
Patch 13632743 – DATABASE PATCH SET UPDATE 10.2.0.5.7 (INCLUDES CPU APR2012)
Patch 13343471 – DATABASE PATCH SET UPDATE 10.2.0.5.6 (INCLUDES CPU JAN2012)
Patch 12827745 – DATABASE PSU 10.2.0.5.5 (INCLUDES CPUOCT2011)
Patch 12419392 – DATABASE PSU 10.2.0.5.4 (INCLUDES CPUJUL2011)
Patch 11724962 – DATABASE PSU 10.2.0.5.3 (INCLUDES CPUAPR2011)
Patch 9952245 – TRACKING BUG FOR 10.2.0.5.2 CRS PSU
Patch 10248542 – DATABASE PSU 10.2.0.5.2 (INCLUDES CPUJAN2011)
Patch 9952230 – DATABASE PSU 10.2.0.5.1 (INCLUDES CPUOCT2010)

Patches for 10.2.0.4
Patch 16619897 – DATABASE PSU 10.2.0.4.17 (REQUIRES PRE-REQUISITE 10.2.0.4.4|INCLUDES CPUJUL2013)
Patch 16056269 – DATABASE PSU 10.2.0.4.16 (REQUIRES PRE-REQUISITE 10.2.0.4.4|INCLUDES CPUAPR2013)
Patch 14736542 – DATABASE PSU 10.2.0.4.15 (REQUIRES PRE-REQUISITE 10.2.0.4.4|INCLUDES CPUJAN2013)
Patch 14275630 – DATABASE PSU 10.2.0.4.14 (REQUIRES PRE-REQUISITE 10.2.0.4.4|INCLUDES CPUOCT2012)
Patch 13923851 – DATABASE PSU 10.2.0.4.13 (REQUIRES PRE-REQUISITE 10.2.0.4.4|INCLUDES CPUJUL2012)
Patch 12879933 – DATABASE PSU 10.2.0.4.12 (REQUIRES PRE-REQUISITE 10.2.0.4.4|INCLUDES CPUAPR2012)
Patch 12879929 – DATABASE PATCH SET UPDATE 10.2.0.4.11 (PRE-REQ 10.2.0.4.4|INCLUDES CPUJAN2012)
Patch 12827778 – DATABASE PSU 10.2.0.4.10 (REQUIRES PRE-REQUISITE 10.2.0.4.4|INCLUDES CPUOCT2011)
Patch 12419397 – DATABASE PSU 10.2.0.4.9 (REQUIRES PRE-REQUISITE 10.2.0.4.4|INCLUDES CPUJUL2011)
Patch 11724977 – DATABASE PSU 10.2.0.4.8 (REQUIRES PRE-REQUISITE 10.2.0.4.4|INCLUDES CPUAPR2011)
Patch 10248636 – DATABASE PSU 10.2.0.4.7 (REQUIRES PRE-REQUISITE 10.2.0.4.4|INCLUDES CPUJAN2011)
Patch 9952234 – DATABASE PSU 10.2.0.4.6 (REQUIRES PRE-REQUISITE 10.2.0.4.4|INCLUDES CPUOCT2010)
Patch 9654991 – DATABASE PSU 10.2.0.4.5 (REQUIRES PRE-REQUISITE 10.2.0.4.4|INCLUDES CPUJUL2010)
Patch 9294403 – TRACKING BUG FOR THE 10.2.0.4.4 CRS PSU
Patch 9352164 – DATABASE PSU 10.2.0.4.4 (INCLUDES CPUAPR2010)
Patch 9119284 – DATABASE PSU 10.2.0.4.3 (INCLUDES CPUJAN2010)
Patch 8833280 – DATABASE PSU 10.2.0.4.2 (INCLUDES CPUOCT2009)
Patch 8576156 – DATABASE PSU 10.2.0.4.1 (INCLUDES CPUJUL2009)