OPatch Auto Failing to Apply Datapatch for a Removed Database

Hi all,

So, just a few days ago, during a client support, crossed the following case.

A few database creations initially failed with DBCA due to other issues, but it seems the DBCA didn’t cleread all creation steps after the failure and theoretical rollback.

As a consequence, whenever running DBCA with GUI client was seeing an old database. When trying to remove it with DBCA, the removal fail as the database can’t be brought up (creation has failed, remember?). Ok, we can leave with it, right?

Yes until we reached the point where Opatchauto failed with the following:

Verifying SQL patch applicability on home /u01/app/oracle/product/19c/db

"/bin/sh -c 'cd /u01/app/oracle/product/19c/db; ORACLE_HOME=/u01/app/oracle/product/19c/db ORACLE_SID=DB1 /u01/app/oracle/product/19c/db/OPatch/datapatch -prereq -verbose'" command failed with errors. Please refer to logs for more details. SQL changes, if any, can be analyzed by manually retrying the same command.

The reason? See the complete log about the failing step:

Executing command as oracle: 
 /bin/sh -c 'cd /u01/app/oracle/product/19c/db;ORACLE_HOME=/u01/app/oracle/product/19c/db ORACLE_SID=DB1 /u01/app/oracle/product/19c/db/OPatch/datapatch -verbose'
2020-09-02 16:26:56,362 INFO  [438] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - COMMAND Looks like this: /bin/sh -c 'cd /u01/app/oracle/product/19c/db;ORACLE_HOME=/u01/app/oracle/product/19c/db ORACLE_SID=DB1 /u01/app/oracle/product/19c/db/OPatch/datapatch -verbose'
2020-09-02 16:26:57,662 INFO  [438] com.oracle.glcm.patch.auto.db.product.executor.GISystemCall - Is retry required=false
2020-09-02 16:26:57,662 INFO  [438] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - status: 1
2020-09-02 16:26:57,662 INFO  [438] com.oracle.glcm.patch.auto.db.product.executor.PatchingStepExecutor - 
COMMAND EXECUTION FAILURE :
SQL Patching tool version 19.8.0.0.0 Production on Wed Sep  2 16:26:57 2020
Copyright (c) 2012, 2020, Oracle.  All rights reserved.

Log file for this invocation: /u01/app/oracle/base/cfgtoollogs/sqlpatch/sqlpatch_25218_2020_09_02_16_26_57/sqlpatch_invocation.log

Connecting to database...
Error: prereq checks failed!
Database connect failed with: ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory
Additional information: 4376
Additional information: 1275019259 (DBD ERROR: OCISessionBegin)

Please refer to MOS Note 1609718.1 and/or the invocation log
/u01/app/oracle/base/cfgtoollogs/sqlpatch/sqlpatch_25218_2020_09_02_16_26_57/sqlpatch_invocation.log
for information on how to resolve the above errors.

SQL Patching tool complete on Wed Sep  2 16:26:57 2020

Clearly, the database is still in place.
As per MOS (ORA-01078 Can Not Delete Database From Dbca on Linux (Doc ID 1559634.1)) -> See the /etc/oratab!

Thing is, the doesn’t have the DB1 line. Also, all related files, logs directory structure, passwd, init, etc… all wiped out. What else?

Here is goes what seems Oracle forgot to tell:

[oracle@PRODB01 dbca]$ srvctl status database -d DB1
Database is not running.
[oracle@DMSDB1PA dbca]$ srvctl config database -d DB1
Database unique name: DB1
Database name: DB1
Oracle home: /u01/app/oracle/product/19c/db
Oracle user: oracle
Spfile: 
Password file: 
Domain: 
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Disk Groups: DATA
Services: 
OSDBA group: oinstall
OSOPER group: 
Database instance: DB1
Ohhh, that took me a while to realize, this was a Standalone server. Once understood, the fix is straight forward:
[oracle@PRODB01 dbca]$ srvctl remove database -d DB1
Remove the database dms? (y/[n]) y
[oracle@PRODB01 dbca]$

I hope this can buy you some minutes of MOS, in case you are googling it first.
Or save you in case you gave up MOS already.

Cheers!

Advise: Setting underScore/Hidden Parameters

Please be aware no hidden parameters (parameters starting with “_”) should not be set unless recommended by Oracle explicitly or in case of a perfect match to the case.

This is explained on MOS Best Practices and Guidelines for Setting underScore/Hidden Parameter to Workaround and/or Resolve Customers Problems (Doc ID 2524119.1)

So, in case of any diagnostics you have to lead to setting it, SUGGESTED:

  • Get the explicit approval/agreement from Oracle.
  • Include evidence on the root cause for this setting. A comment should work, something like:
alter system set "_replace_virtual_columns"=false Comment= 'Workaround Doc ID 1512347.1' SID= '*' scope= both;

Cheers!

Oracle Patching with OPlan

Everyone that I’ve worked with knows that I don’t like patching (and sometimes I try to imagine who does), but they are necessary to corrects bugs and improve the Oracle software stability.

When you have a single node server with one database, the patch planing is no brainer but when you have a RAC with multiple nodes, different Oracle homes and so on, the planning and preparations start to get more complex and it is easy to miss or overlook a step in the planning which can lead to issues during your patching.

So to help me with all that I use oplan. Oplan is a tool which comes with along OPatch and you can get its latest version in patch 6880880

More informations on oplan can be found here: Oracle Software Patching with OPLAN (Doc ID 1306814.1)

OK, so what do I used it most for?

Generating the apply patching steps, which are very in handy:

$ORACLE_HOME/OPatch/oplan/oplan generateApplySteps <bundle patch location>

And my favorite, rollback steps, which I have done more times that I would like to admit:

$ORACLE_HOME/OPatch/oplan/oplan generateRollbackSteps <bundle patch location>

Also as rollback, I do tar of the oracle binaries being patched prior as there times even the rollback did not work :-/

Both files will be created under the directory below and you will see an html and text files.

$ORACLE_HOME/cfgtoollogs/oplan/<TimeStamp>/

This process is to help you organise your steps, read it through prior executing to make sure it makes sense in your environment

Oplan has its limitations, from the Oracle note which I mentioned above:

Data Guard configurations are not supported.
OPlan can be used to create patch plans for Oracle home's running Oracle Data Guard configurations, but OPlan does not consider such an environment usable as 'Data Guard Standby-First Patch Apply' alternative. See the following for additional information on 'Data Guard Standby-First Patch Apply'

<Document 1265700.1> Oracle Patch Assurance - Data Guard Standby-First Patch Apply

Shared Oracle Home Configurations are not supported.

Single Instance Databases running in the same configuration are not supported

Even so I would still use it as it generates a plan based on your target environment adding more information that you would need to do manually if you were only to read the README files from the patching

Hope it helps.

Thanks and until next time

Elisson Almeida

Getting Oracle version – new utility on 18c oraversion

While scripting on an environment with different Oracle versions, I often needed to get the Oracle version for the target database as usually the SQL to be executed is version depended.

I used different methods to grab the database  version some you can see below:

SQL> select 
substr(
substr(
banner, 
instr(banner,' ',-(length(banner)-instr(banner,'.')),1)+1, 
instr(banner,' ',+instr(banner,'.'),1)-instr(banner,' ',-(length(banner)-instr(banner,'.')),1)-1
),
1, 
instr(
substr(
banner, 
instr(banner,' ',-(length(banner)-instr(banner,'.')),1)+1,
instr(banner,' ',+instr(banner,'.'),1)-instr(banner,' ',-(length(banner)-instr(banner,'.')),1)-1
),
'.'
)-1
) version
from v$version
where rownum = 1
; 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

VERSION
--------------------------------------------------------------------------------
18

Or like this

SQL> select substr(version,1,instr(version,'.')-1) version from product_component_version where product like 'Oracle%';

VERSION
--------------------------------------------------------------------------------
18

But on Oracle 18c there is a new utility called oraversion which you can get the same result as the above queries.

[oracle@server01 ~]$ oraversion
This program prints release version information.
These are its possible arguments:
-compositeVersion: Print the full version number: a.b.c.d.e.
-baseVersion: Print the base version number: a.0.0.0.0.
-majorVersion: Print the major version number: a.
-buildStamp: Print the date/time associated with the build.
-buildDescription: Print a description of the build.
-help: Print this message.
[oracle@server1 ~]$

[oracle@server01 ~]$ oraversion -majorVersion
18
[oracle@server01 ~]$

This could be somewhat useful but I though it was worth sharing.

Until next time.

Elisson Almeida

Stop Migrating Databases to 12c!

If you are starting an upgrade plan, do it directly to 19c. Why?

Well, first let’s say that 19c is “equal” to 12.2.0.3 on previous version model, while 18c is 12.2.0.2, after latest 12.2.0.1. So, thinking reasonably, you would already plan for latest PSU on the future version, right? So this is 19c.

But not only this. It’s well known that Oracle’s Terminal Patchset is the Long Term Release for each version. And for 12.2 family this is 19c.

The image below (From Doc ID 742060.1) should help on clarifying this:

versions

In summary, since August 1st 2019:
– 11.2.0.4 is already on Extended Support (20% more expensive unless you have ULA)
– 12.1 family is already on Extended Support (20% more expensive unless you have ULA)

Which only leave us with 12.2 Family. Now I ask you, why jump to any “PSU” other than the final one?
Also, support on 12.2.0.1 goes only up to Nov 2020 and for 18c (12.2.0.2) only to Jun 2021, while 19c goes up to 2026 with ES/ULA!

I’m not even raising the great improvements we have on newer versions (if you are in 11.2, shame on you, you are basically 6 years behind the world technology wise).
Myself , I already migrated several databases on different clients to 19c and it’s running preeeetty fine. So trust on it!

Ok, so to migrate to 19c? (Some tips):
– To migrate directly to 19c you need to be on supported terminal patch set (11.2.0.4, 12.1.0.2) or on 12.2 family (12.2.0.1, 18c).
– Get in touch and use AutoUpgrade mechanism (you’ll love it!) – AutoUpgrade Tool (Doc ID 2485457.1)
– Check Upgrade Guide: https://docs.oracle.com/en/database/oracle/oracle-database/19/upgrd/index.html
– Have a look on Mike Dietrich blog for news: https://mikedietrichde.com/
– Test it before doing in prod.

Additional Reference:
Release Schedule of Current Database Releases (Doc ID 742060.1)
Very nice and complete post (in portuguese) on this topic from Ricardo Portilho.

I like very much the table below as well:

NewScreenshot 2019-08-13 às 15.11.34

See you!

Invalid system views after upgrading from 12c to 18c – X_$KGLOB and X_$KGLCURSOR

I was working on a client upgrade from 12c to 18c. The upgrade was done manually. After the dbupgrade 2 system views got invalid, X_$KGLCURSOR and X_$KGLOB

OWNER OBJECT_NAME OBJECT_TYPE STATUS
------------ -------------------------------------------------- ---------------------------------------- -------
SYS X_$KGLOB VIEW INVALID
SYS X_$KGLCURSOR VIEW INVALID

Man system objects invalid after an upgrade is never good.

Troubleshooting, I got the view text and tried to execute its sql

SQL> select TEXT from dba_Views where view_name = 'X_$KGLOB';

TEXT
------------------------------------------------------------------------------------------------------------------------------------------------------
SELECT "ADDR","INDX","INST_ID","CON_ID","KGLHDADR","KGLHDPAR","KGLHDCLT","KGLNAOWN","KGLNAOBJ","KGLFNOBJ","KGLNADLK","KGLNAHSH","KGLNAHSV","KGLNATIM",
"KGLNAPTM","KGLHDNSP","KGLHDNSD","KGLHDLMD","KGLHDPMD","KGLHDFLG","KGLHDOBJ","KGLHDLDC","KGLHDIVC","KGLHDEXC","KGLHDLKC","KGLHDKMK","KGLHDDMK","KGLHDA
MK","KGLOBFLG","KGLOBSTA","KGLOBTYP","KGLOBTYD","KGLOBHMK","KGLOBHS0","KGLOBHS1","KGLOBHS2","KGLOBHS3","KGLOBHS4","KGLOBHS5","KGLOBHS6","KGLOBHS7","KG
LOBHD0","KGLOBHD1","KGLOBHD2","KGLOBHD3","KGLOBHD4","KGLOBHD5","KGLOBHD6","KGLOBHD7","KGLOBPC0","KGLOBPC6","KGLOBTP0","KGLOBT00","KGLOBT01","KGLOBT02"
,"KGLOBT03","KGLOBT04","KGLOBT05","KGLOBT35","KGLOBT06","KGLOBT07","KGLOBT08","KGLOBT09","KGLOBT10","KGLOBT11","KGLOBT12","KGLOBT13","KGLOBT14","KGLOB
T15","KGLOBT16","KGLOBT17","KGLOBT18","KGLOBT19","KGLOBT20","KGLOBT21","KGLOBT22","KGLOBT23","KGLOBT24","KGLOBT25","KGLOBT26","KGLOBT28","KGLOBT29","K
GLOBT30","KGLOBT31","KGLOBT27","KGLOBT32","KGLOBT33","KGLOBWAP","KGLOBWCC","KGLOBWCL","KGLOBWUI","KGLOBWDW","KGLOBT42","KGLOBT43","KGLOBT44","KGLOBT45
","KGLOBT46","KGLOBT47","KGLOBT49","KGLOBT50","KGLOBT52","KGLOBT53","KGLOBTL0","KGLOBTL1","KGLOBTS0","KGLOBTS1","KGLOBTN0","KGLOBTN1","KGLOBTN2","KGLO
BTN3","KGLOBTN4","KGLOBTN5","KGLOBTS2","KGLOBTS3","KGLOBTS5","KGLOBTT0","KGLOBCCE","KGLOBCCEH","KGLOBCLA","KGLOBCLC","KGLOBCCC","KGLOBTS4","KGLOBCBCA"
,"KGLOBT48","KGLOBDSO","KGLOBDEX","KGLOBDPX","KGLOBDLD","KGLOBDIV","KGLOBDPS","KGLOBDDR","KGLOBDDW","KGLOBDBF","KGLOBDRO","KGLOBDCP","KGLOBDEL","KGLOB
DFT","KGLOBDEF","KGLOBDUI","KGLOBDCL","KGLOBDAP","KGLOBDCC","KGLOBDPL","KGLOBDJV","KGLOBACS","KGLOBTS6","KGLOBTS7","KGLOBT54","KGLOBT55","KGLOBT56","K
GLOBT57","KGLOBDCO","KGLOBDCI","KGLOBDRR","KGLOBDRB","KGLOBDWR","KGLOBDWB","KGLOBT58","KGLOBDOR","KGLHDMTX","KGLHDMVL","KGLHDMSP","KGLHDMGT","KGLHDDMT
X","KGLHDDVL","KGLHDDSP","KGLHDDGT","KGLHDBID","KGLHDBMTX","KGLHDBVL","KGLHDBSP","KGLHDBGT","KGLOBT59","KGLOBDCU","KGLOBPROP","KGLNACON","KGLNAO1N","K
GLHDMAR","KGLMPFLG","KGLOBAQP","KGLOBT60","KGLOBT61","KGLIMSCANS","KGLIMBYTESUNCOMP","KGLIMBYTESINMEM" FROM X$KGLOB


1 row selected.
SQL> SELECT "ADDR","INDX","INST_ID","CON_ID","KGLHDADR","KGLHDPAR","KGLHDCLT","KGLNAOWN","KGLNAOBJ","KGLFNOBJ","KGLNADLK","KGLNAHSH","KGLNAHSV","KGLNATIM","KGLNAPTM","KGLHDNSP","KGLHDNSD","KGLHDLMD","KGLHDPMD","KGLHDFLG","KGLHDOBJ","KGLHDLDC","KGLHDIVC","KGLHDEXC","KGLHDLKC","KGLHDKMK","KGLHDDMK","KGLHDAMK","KGLOBFLG","KGLOBSTA","KGLOBTYP","KGLOBTYD","KGLOBHMK","KGLOBHS0","KGLOBHS1","KGLOBHS2","KGLOBHS3","KGLOBHS4","KGLOBHS5","KGLOBHS6","KGLOBHS7","KGLOBHD0","KGLOBHD1","KGLOBHD2","KGLOBHD3","KGLOBHD4","KGLOBHD5","KGLOBHD6","KGLOBHD7","KGLOBPC0","KGLOBPC6","KGLOBTP0","KGLOBT00","KGLOBT01","KGLOBT02","KGLOBT03","KGLOBT04","KGLOBT05","KGLOBT35","KGLOBT06","KGLOBT07","KGLOBT08","KGLOBT09","KGLOBT10","KGLOBT11","KGLOBT12","KGLOBT13","KGLOBT14","KGLOBT15","KGLOBT16","KGLOBT17","KGLOBT18","KGLOBT19","KGLOBT20","KGLOBT21","KGLOBT22","KGLOBT23","KGLOBT24","KGLOBT25","KGLOBT26","KGLOBT28","KGLOBT29","KGLOBT30","KGLOBT31","KGLOBT27","KGLOBT32","KGLOBT33","KGLOBWAP","KGLOBWCC","KGLOBWCL","KGLOBWUI","KGLOBWDW","KGLOBT42","KGLOBT43","KGLOBT44","KGLOBT45","KGLOBT46","KGLOBT47","KGLOBT49","KGLOBT50","KGLOBT52","KGLOBT53","KGLOBTL0","KGLOBTL1","KGLOBTS0","KGLOBTS1","KGLOBTN0","KGLOBTN1","KGLOBTN2","KGLOBTN3","KGLOBTN4","KGLOBTN5","KGLOBTS2","KGLOBTS3","KGLOBTS5","KGLOBTT0","KGLOBCCE","KGLOBCCEH","KGLOBCLA","KGLOBCLC","KGLOBCCC","KGLOBTS4","KGLOBCBCA","KGLOBT48","KGLOBDSO","KGLOBDEX","KGLOBDPX","KGLOBDLD","KGLOBDIV","KGLOBDPS","KGLOBDDR","KGLOBDDW","KGLOBDBF","KGLOBDRO","KGLOBDCP","KGLOBDEL","KGLOBDFT","KGLOBDEF","KGLOBDUI","KGLOBDCL","KGLOBDAP","KGLOBDCC","KGLOBDPL","KGLOBDJV","KGLOBACS","KGLOBTS6","KGLOBTS7","KGLOBT54","KGLOBT55","KGLOBT56","KGLOBT57","KGLOBDCO","KGLOBDCI","KGLOBDRR","KGLOBDRB","KGLOBDWR","KGLOBDWB","KGLOBT58","KGLOBDOR","KGLHDMTX","KGLHDMVL","KGLHDMSP","KGLHDMGT","KGLHDDMTX","KGLHDDVL","KGLHDDSP","KGLHDDGT","KGLHDBID","KGLHDBMTX","KGLHDBVL","KGLHDBSP","KGLHDBGT","KGLOBT59","KGLOBDCU","KGLOBPROP","KGLNACON","KGLNAO1N","KGLHDMAR","KGLMPFLG","KGLOBAQP","KGLOBT60","KGLOBT61","KGLIMSCANS","KGLIMBYTESUNCOMP","KGLIMBYTESINMEM" FROM X$KGLOB;
SELECT "ADDR","INDX","INST_ID","CON_ID","KGLHDADR","KGLHDPAR","KGLHDCLT","KGLNAOWN","KGLNAOBJ","KGLFNOBJ","KGLNADLK","KGLNAHSH","KGLNAHSV","KGLNATIM","KGLNAPTM","KGLHDNSP","KGLHDNSD","KGLHDLMD","KGLHDPMD","KGLHDFLG","KGLHDOBJ","KGLHDLDC","KGLHDIVC","KGLHDEXC","KGLHDLKC","KGLHDKMK","KGLHDDMK","KGLHDAMK","KGLOBFLG","KGLOBSTA","KGLOBTYP","KGLOBTYD","KGLOBHMK","KGLOBHS0","KGLOBHS1","KGLOBHS2","KGLOBHS3","KGLOBHS4","KGLOBHS5","KGLOBHS6","KGLOBHS7","KGLOBHD0","KGLOBHD1","KGLOBHD2","KGLOBHD3","KGLOBHD4","KGLOBHD5","KGLOBHD6","KGLOBHD7","KGLOBPC0","KGLOBPC6","KGLOBTP0","KGLOBT00","KGLOBT01","KGLOBT02","KGLOBT03","KGLOBT04","KGLOBT05","KGLOBT35","KGLOBT06","KGLOBT07","KGLOBT08","KGLOBT09","KGLOBT10","KGLOBT11","KGLOBT12","KGLOBT13","KGLOBT14","KGLOBT15","KGLOBT16","KGLOBT17","KGLOBT18","KGLOBT19","KGLOBT20","KGLOBT21","KGLOBT22","KGLOBT23","KGLOBT24","KGLOBT25","KGLOBT26","KGLOBT28","KGLOBT29","KGLOBT30","KGLOBT31","KGLOBT27","KGLOBT32","KGLOBT33","KGLOBWAP","KGLOBWCC","KGLOBWCL","KGLOBWUI","KGLOBWDW","KGLOBT42","KGLOBT43","KGLOBT44","KGLOBT45","KGLOBT46","KGLOBT47","KGLOBT49","KGLOBT50","KGLOBT52","KGLOBT53","KGLOBTL0","KGLOBTL1","KGLOBTS0","KGLOBTS1","KGLOBTN0","KGLOBTN1","KGLOBTN2","KGLOBTN3","KGLOBTN4","KGLOBTN5","KGLOBTS2","KGLOBTS3","KGLOBTS5","KGLOBTT0","KGLOBCCE","KGLOBCCEH","KGLOBCLA","KGLOBCLC","KGLOBCCC","KGLOBTS4","KGLOBCBCA","KGLOBT48","KGLOBDSO","KGLOBDEX","KGLOBDPX","KGLOBDLD","KGLOBDIV","KGLOBDPS","KGLOBDDR","KGLOBDDW","KGLOBDBF","KGLOBDRO","KGLOBDCP","KGLOBDEL","KGLOBDFT","KGLOBDEF","KGLOBDUI","KGLOBDCL","KGLOBDAP","KGLOBDCC","KGLOBDPL","KGLOBDJV","KGLOBACS","KGLOBTS6","KGLOBTS7","KGLOBT54","KGLOBT55","KGLOBT56","KGLOBT57","KGLOBDCO","KGLOBDCI","KGLOBDRR","KGLOBDRB","KGLOBDWR","KGLOBDWB","KGLOBT58","KGLOBDOR","KGLHDMTX","KGLHDMVL","KGLHDMSP","KGLHDMGT","KGLHDDMTX","KGLHDDVL","KGLHDDSP","KGLHDDGT","KGLHDBID","KGLHDBMTX","KGLHDBVL","KGLHDBSP","KGLHDBGT","KGLOBT59","KGLOBDCU","KGLOBPROP","KGLNACON","KGLNAO1N","KGLHDMAR","KGLMPFLG","KGLOBAQP","KGLOBT60","KGLOBT61","KGLIMSCANS","KGLIMBYTESUNCOMP","KGLIMBYTESINMEM" FROM X$KGLOB
*
ERROR at line 1:
ORA-00904: "KGLOBT24": invalid identifier
SQL> select TEXT from dba_Views where view_name = 'X_$KGLCURSOR'

TEXT
------------------------------------------------------------------------------------------------------------------------------------------------------
SELECT "ADDR","INDX","INST_ID","CON_ID","KGLHDADR","KGLHDPAR","KGLHDCLT","KGLNAOWN","KGLNAOBJ","KGLFNOBJ","KGLNADLK","KGLNAHSH","KGLNAHSV","KGLNATIM",
"KGLNAPTM","KGLHDNSP","KGLHDNSD","KGLHDLMD","KGLHDPMD","KGLHDFLG","KGLHDOBJ","KGLHDLDC","KGLHDIVC","KGLHDEXC","KGLHDLKC","KGLHDKMK","KGLHDDMK","KGLHDA
MK","KGLOBFLG","KGLOBSTA","KGLOBTYP","KGLOBTYD","KGLOBHMK","KGLOBHS0","KGLOBHS1","KGLOBHS2","KGLOBHS3","KGLOBHS4","KGLOBHS5","KGLOBHS6","KGLOBHS7","KG
LOBHD0","KGLOBHD1","KGLOBHD2","KGLOBHD3","KGLOBHD4","KGLOBHD5","KGLOBHD6","KGLOBHD7","KGLOBPC0","KGLOBPC6","KGLOBTP0","KGLOBT00","KGLOBT01","KGLOBT02"
,"KGLOBT03","KGLOBT04","KGLOBT05","KGLOBT35","KGLOBT06","KGLOBT07","KGLOBT08","KGLOBT09","KGLOBT10","KGLOBT11","KGLOBT12","KGLOBT13","KGLOBT14","KGLOB
T15","KGLOBT16","KGLOBT17","KGLOBT18","KGLOBT19","KGLOBT20","KGLOBT21","KGLOBT22","KGLOBT23","KGLOBT24","KGLOBT25","KGLOBT26","KGLOBT28","KGLOBT29","K
GLOBT30","KGLOBT31","KGLOBT27","KGLOBT32","KGLOBT33","KGLOBWAP","KGLOBWCC","KGLOBWCL","KGLOBWUI","KGLOBWDW","KGLOBT42","KGLOBT43","KGLOBT44","KGLOBT45
","KGLOBT46","KGLOBT47","KGLOBT49","KGLOBT50","KGLOBT52","KGLOBT53","KGLOBTL0","KGLOBTL1","KGLOBTS0","KGLOBTS1","KGLOBTN0","KGLOBTN1","KGLOBTN2","KGLO
BTN3","KGLOBTN4","KGLOBTN5","KGLOBTS2","KGLOBTS3","KGLOBTS5","KGLOBTT0","KGLOBCCE","KGLOBCCEH","KGLOBCLA","KGLOBCLC","KGLOBCCC","KGLOBTS4","KGLOBCBCA"
,"KGLOBT48","KGLOBDSO","KGLOBDEX","KGLOBDPX","KGLOBDLD","KGLOBDIV","KGLOBDPS","KGLOBDDR","KGLOBDDW","KGLOBDBF","KGLOBDRO","KGLOBDCP","KGLOBDEL","KGLOB
DFT","KGLOBDEF","KGLOBDUI","KGLOBDCL","KGLOBDAP","KGLOBDCC","KGLOBDPL","KGLOBDJV","KGLOBACS","KGLOBTS6","KGLOBTS7","KGLOBT54","KGLOBT55","KGLOBT56","K
GLOBT57","KGLOBDCO","KGLOBDCI","KGLOBDRR","KGLOBDRB","KGLOBDWR","KGLOBDWB","KGLOBT58","KGLOBDOR","KGLHDMTX","KGLHDMVL","KGLHDMSP","KGLHDMGT","KGLHDDMT
X","KGLHDDVL","KGLHDDSP","KGLHDDGT","KGLHDBID","KGLHDBMTX","KGLHDBVL","KGLHDBSP","KGLHDBGT","KGLOBT59","KGLOBDCU","KGLOBPROP","KGLNACON","KGLNAO1N","K
GLHDMAR","KGLMPFLG","KGLOBAQP","KGLOBT60","KGLOBT61","KGLIMSCANS","KGLIMBYTESUNCOMP","KGLIMBYTESINMEM" FROM X$KGLCURSOR


1 row selected.
SQL> SELECT "ADDR","INDX","INST_ID","CON_ID","KGLHDADR","KGLHDPAR","KGLHDCLT","KGLNAOWN","KGLNAOBJ","KGLFNOBJ","KGLNADLK","KGLNAHSH","KGLNAHSV","KGLNATIM",
"KGLNAPTM","KGLHDNSP","KGLHDNSD","KGLHDLMD","KGLHDPMD","KGLHDFLG","KGLHDOBJ","KGLHDLDC","KGLHDIVC","KGLHDEXC","KGLHDLKC","KGLHDKMK","KGLHDDMK","KGLHDA
MK","KGLOBFLG","KGLOBSTA","KGLOBTYP","KGLOBTYD","KGLOBHMK","KGLOBHS0","KGLOBHS1","KGLOBHS2","KGLOBHS3","KGLOBHS4","KGLOBHS5","KGLOBHS6","KGLOBHS7","KG
LOBHD0","KGLOBHD1","KGLOBHD2","KGLOBHD3","KGLOBHD4","KGLOBHD5","KGLOBHD6","KGLOBHD7","KGLOBPC0","KGLOBPC6","KGLOBTP0","KGLOBT00","KGLOBT01","KGLOBT02"
,"KGLOBT03","KGLOBT04","KGLOBT05","KGLOBT35","KGLOBT06","KGLOBT07","KGLOBT08","KGLOBT09","KGLOBT10","KGLOBT11","KGLOBT12","KGLOBT13","KGLOBT14","KGLOB
T15","KGLOBT16","KGLOBT17","KGLOBT18","KGLOBT19","KGLOBT20","KGLOBT21","KGLOBT22","KGLOBT23","KGLOBT24","KGLOBT25","KGLOBT26","KGLOBT28","KGLOBT29","K
GLOBT30","KGLOBT31","KGLOBT27","KGLOBT32","KGLOBT33","KGLOBWAP","KGLOBWCC","KGLOBWCL","KGLOBWUI","KGLOBWDW","KGLOBT42","KGLOBT43","KGLOBT44","KGLOBT45
","KGLOBT46","KGLOBT47","KGLOBT49","KGLOBT50","KGLOBT52","KGLOBT53","KGLOBTL0","KGLOBTL1","KGLOBTS0","KGLOBTS1","KGLOBTN0","KGLOBTN1","KGLOBTN2","KGLO
BTN3","KGLOBTN4","KGLOBTN5","KGLOBTS2","KGLOBTS3","KGLOBTS5","KGLOBTT0","KGLOBCCE","KGLOBCCEH","KGLOBCLA","KGLOBCLC","KGLOBCCC","KGLOBTS4","KGLOBCBCA"
,"KGLOBT48","KGLOBDSO","KGLOBDEX","KGLOBDPX","KGLOBDLD","KGLOBDIV","KGLOBDPS","KGLOBDDR","KGLOBDDW","KGLOBDBF","KGLOBDRO","KGLOBDCP","KGLOBDEL","KGLOB
DFT","KGLOBDEF","KGLOBDUI","KGLOBDCL","KGLOBDAP","KGLOBDCC","KGLOBDPL","KGLOBDJV","KGLOBACS","KGLOBTS6","KGLOBTS7","KGLOBT54","KGLOBT55","KGLOBT56","K
GLOBT57","KGLOBDCO","KGLOBDCI","KGLOBDRR","KGLOBDRB","KGLOBDWR","KGLOBDWB","KGLOBT58","KGLOBDOR","KGLHDMTX","KGLHDMVL","KGLHDMSP","KGLHDMGT","KGLHDDMT
X","KGLHDDVL","KGLHDDSP","KGLHDDGT","KGLHDBID","KGLHDBMTX","KGLHDBVL","KGLHDBSP","KGLHDBGT","KGLOBT59","KGLOBDCU","KGLOBPROP","KGLNACON","KGLNAO1N","K
GLHDMAR","KGLMPFLG","KGLOBAQP","KGLOBT60","KGLOBT61","KGLIMSCANS","KGLIMBYTESUNCOMP","KGLIMBYTESINMEM" FROM X$KGLCURSOR
2 3 4 5 6 7 8 9 10 11 12 13 14 15 
SQL> /
X","KGLHDDVL","KGLHDDSP","KGLHDDGT","KGLHDBID","KGLHDBMTX","KGLHDBVL","KGLHDBSP","KGLHDBGT","KGLOBT59","KGLOBDCU","KGLOBPROP","KGLNACON","KGLNAO1N","K
*
ERROR at line 13:
ORA-00904: "KGLHDMAR": invalid identifier

Describing the source tables used by the views, the columns that generated the error do not exist there. What?!

Looking up MOS – found this note, Invalid x_$ Objects After Upgrade (Doc ID 361757.1)

Although this note does not mention these views explicitly there is a part of that says that

"This is NOT a dictionary corruption, but the result of an unsupported operation.

.....

These x_$ views do not compile because they are based on x$ tables which have changed (columns dropped) and/or do not exist (X$ table dropped).

New (default) 10g & 11g databases do not have views with the prefix x_$.

Please note that this solution can be applied to any and all x_$ views found invalid after upgrade/migration.
These tables are built on x$ tables in the previous release and the same rules still apply."

So these are save to ignore or drop them.

Hope this helps.

Elisson Almeida

Waiting for ASM to startup after upgrading to Oracle 18c

Hi,
Something strange happened to me and I would like to share with you. I was upgrading one of my Virtual Machine which was an Oracle Restart – SIHA,  running 11.2.0.4 to 18c on Oracle Linux 6.9.

The upgrade went fine using the GUI no issues nor strange log messages, but the issue happened when I rebooted the VM. For some reason (which I don’t knew) the startup was hanging when trying to start HAS and then it outputted the error below:

PRCR-1070 :Failed to check if resource ora.asm is registered
CRS-0184 : Cannot communicate with the CRS daemon
Waiting for ASM to startup

Looking up on MOS I found this note – Oracle Linux 6 server hangs on boot with error: Waiting for ASM to startup (Doc ID 2495023.1)

The startup was hanging waiting on acfssihamount but as ASM was not up yet and it could not start.

To fix it you need to boot OS in rescue mode and execute

chroot /mnt/sysimage

Once you do that you will be able to change the startup configuration settings using chkconfig and disable acfssihamount

chkconfig --list acfssihamount
chkconfig acfssihamount off
chkconfig --list acfssihamount

After this exit out the rescue mode and the system should boot normally.

Hope it helps!

Elisson Almeida

DO NOT uninstall Database Vault from a CDB!

Hi all,

This is just a note for you that are now fully into database upgrades. As you may know, the Database Vault (DV) option could be reported as not installed in source when converting non-CDB to PDB, and the target CDB has the option installed. However, be aware that is not supported to remove DV option in multitenant (singletenant included) environments.

From Doc ID 2380950.1:

IMPORTANT:  Never run dvremov.sql on a container database (CDB) / Multitenant environment.  Doing so will cause serious problems when upgrading the PDBs.

 

In case you run dvremov.sql in a 12.1 multi-tenant DB, then the only option you might have is to restore the database to the state before running dvremov.sql. Assuming you have this backup, of course.

So stay sharp! 😀

Some references:
How To uninstall / install Database Vault in a 12c database ? (Doc ID 2112167.1)
DBA_DV_STATUS Is INVALID (Due to “ORA-00942: Table Or View Does Not Exist”) (Doc ID 2380950.1)

12c Datapatch Failing on ORA-65108: Applying Datapatch Manually

Hi all,

So, in a new 12c environment datapatch failed on the following:

ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-65108: invalid use of a cursor belonging to another container
kpdbaKillPdbSessions: Starting kill.
2019-01-29 07:09:21.486584 :kjcipctxinit(): (pid|psn)=(25|2): initialised and linked pctx 0x00007FFB1A7B49C8 into process list
ORA-00704: bootstrap process failure
ORA-00604: error occurred at recursive SQL level 1
ORA-65108: invalid use of a cursor belonging to another container

Point is that was failing for the PDB$SEED. When upgrading, there are phases where you need the seed opened read-write. But in general you don’t to that yourself.
The scripts to run in each container are called through catcon.pl which, by default, opens the seed read-write and ensures that the initial open mode is restored at the end even in case of error.

So how to manually proceed with this?

1. Apply the datapatch first to other PDBs using the -pds clause:

./datapatch -verbose -pdbs PDB1,PDB2

2. Open PDB$SEED for Oracle Scripts:

SYS@CDB$ROOT SQL> alter session set "_oracle_script"=true;
 Session altered. 
 SYS@CDB$ROOT SQL> alter pluggable database pdb$seed open read write force;
 Pluggable database PDB$SEED altered.

3. Apply the datapatch to the PDB$SEED

$ORACLE_HOME/OPatch/datapatch -verbose -pdbs ‘PDB$SEED’

4. Close PDB$SEED and Mount it back

 SYS@CDB$ROOT SQL> alter session set container=PDB$SEED
 Session altered. 
 SYS@CDB$ROOT SQL> shutdown immediate
 Pluggable Database closed
++ From Root
 SYS@CDB$ROOT SQL> alter pluggable database PDB$SEED open read only;

Hope it helps, cheers!

ORA-00932 to Export Statistics After 12c Upgrade – Undocumented

Hello All,
I have a client that use to Export and Import Dictionary Statistics using a stats table. However, after upgrading database to 12c, is started to raise error below to export data to a stats table (created in 11g) or even to update stats in this table.

SQL> exec dbms_stats.upgrade_stat_table('GREPORA','MYSTATSTABLE');
DECLARE
*
ERROR at line 1:
ORA-00932: inconsistent datatypes: expected BINARY got NUMBER
ORA-06512: at "SYS.DBMS_STATS", line 40183
ORA-06512: at line 28

SQL> EXEC DBMS_STATS.EXPORT_FIXED_OBJECTS_STATS (stattab => 'MYSTATSTABLE',statid  => 'TEST_MATHEUS',statown => 'GREPORA' );
BEGIN DBMS_STATS.EXPORT_FIXED_OBJECTS_STATS (stattab => 'MYSTATSTABLE',statid  => 'TEST_MATHEUS',statown => 'GREPORA' ); END;

*
ERROR at line 1:
ORA-00932: inconsistent datatypes: expected BINARY got NUMBER
ORA-06512: at "SYS.DBMS_STATS", line 37085
ORA-06512: at line 1

I reviewed several references (all very interesting, by the way, recommend you to take a look):
– ORA-20002 on Importing Statistics using DBMS_STATS Procedures (Doc ID 740345.1)
– Datapump Export Fails With Ora-00932 (Doc ID 1300803.1)
– Error “ORA-00932 – Inconsistent Datatypes: Expected BINARY Got NUMBER” When Executing the Primavera Gather-Statistics Scripts on an Oracle 12c Database (Doc ID 2111114.1)
https://oracle-base.com/articles/12c/concurrent-statistics-collection-12cr1
– http://www.morganslibrary.org/reference/pkgs/dbms_stats.html
– How To Export and Import Dictionary Statistics (Doc ID 1450820.1)
– Fixed Objects Statistics (GATHER_FIXED_OBJECTS_STATS) Considerations (Doc ID 798257.1)

But no reference found to this specific issue, not even in MOS.

After some tests, I realized that maybe columns order in stats table can be changed in implementation between 11g and 12c. Bingo! The same columns and datatypes are in place, but in different order.

This is not documented in MOS neither on internet/independent blogs, but seems EXPORT% on DBMS_STATS presume stats table in it’s internal code without specified column orders.
As we know, a good development practice is to specify column names when performing any operation. Bad on you, Oracle!

More“ORA-00932 to Export Statistics After 12c Upgrade – Undocumented”