Orphan ASM File Cleanup Script

Hi all,

So I got asked by a client to perform a checking on ASM for orphan files, as they have some frequent create/drop database on this environment, as being a development env.

Also, lots of databases shared the same data diskgroup, so I had to work this out for all databases and also for possible inexistent databases.

Some basic approaches I raised:

1) Locating uncatalloged files in ASM per database.
– Source: https://oraganism.wordpress.com/2012/09/09/orphaned-files-in-asm/
– This approach assumes the files on ASM uncatalogged to any database are the Orphaned ones. Which is a fair assumption.
– But I understand that files can be catalloged and unmonted, which would brake this approach.

2) Listing files in ASM but not in database (v$datafile, v$datafile_copy, v$controlfile, v$tempfile, v$logfile) by database.
– Source: https://oracledba.blogspot.com/2018/11/orphaned-files-in-asm.html
– This seems a fair assumption. Would need to be ran from each database.
– There is not guarantee if this is working properly or not.
– Not clear also if PDB files are included.
– There is another similar one: https://dbaliveblog.wordpress.com/asm-orphaned-file-identification-script/
– Also this one: https://anjo.pt/wp/keyword-oracle/2013/02/26/find-orphan-asm-files/

3) MOS: Query That Can Be Used to Find Orphaned Datafiles on a 12c ASM Instance (Doc ID 2228573.1)
– From MOS, seems the most recommended approach.
– Attention point: PDB$SEED may not be shown as per: PDB$SEED Datafiles Not Appear In CDB_DATA_FILES (Doc ID 1940806.1)
— On 12.1.0.2, recommended to use “EXCLUDE_SEED_CDB_VIEW”. To check if it can be done on session level.

I downloaded and ran referred script on MOS Script to report the list of files stored in ASM and CURRENTLY NOT OPENED (Doc ID 552082.1) on the environment.
But the results didn’t sound correct.

After a while, I ended up building my own script based on all mentioned approaches and it worked very fine.

After approved I dropped all the listed files, freed a several TBs of space and no database affected. So I’d assume it as correct and would really recommend it for you.

So what did I used:

SQL to Check ASM Space per Database:

set pages 350 timing on
col gname form a10
col dbname form a10
col file_type form a16
break on gname skip 2 on dbname skip 1
compute sum label total_db of gb on dbname
compute sum label total_diskg of gb on gname  
SELECT
    gname,
    dbname,
    file_type,
    round(SUM(space)/1024/1024) mb,
    round(SUM(space)/1024/1024/1024) gb,
    COUNT(*) "#FILES"
FROM
    (
        SELECT
            gname,
            regexp_substr(full_alias_path, '[[:alnum:]_]*',1,4) dbname,
            file_type,
            space,
            aname,
            system_created,
            alias_directory
        FROM
            (
                SELECT
                    concat('+'||gname, sys_connect_by_path(aname, '/')) full_alias_path,
                    system_created,
                    alias_directory,
                    file_type,
                    space,
                    level,
                    gname,
                    aname
                FROM
                    (
                        SELECT
                            b.name            gname,
                            a.parent_index    pindex,
                            a.name            aname,
                            a.reference_index rindex ,
                            a.system_created,
                            a.alias_directory,
                            c.type file_type,
                            c.space
                        FROM
                            v$asm_alias a,
                            v$asm_diskgroup b,
                            v$asm_file c
                        WHERE
                            a.group_number = b.group_number
                        AND a.group_number = c.group_number(+)
                        AND a.file_number = c.file_number(+)
                        AND a.file_incarnation = c.incarnation(+) ) START WITH (mod(pindex, power(2, 24))) = 0
                AND rindex IN
                    (
                        SELECT
                            a.reference_index
                        FROM
                            v$asm_alias a,
                            v$asm_diskgroup b
                        WHERE
                            a.group_number = b.group_number
                        AND (
                                mod(a.parent_index, power(2, 24))) = 0
                    ) CONNECT BY prior rindex = pindex )
        WHERE
            NOT file_type IS NULL
            and system_created = 'Y' )
GROUP BY
    gname,
    dbname,
    file_type
ORDER BY
    gname,
    dbname,
    file_type
/

Expected Output:

SQL> @asm_sizebydb

GNAME	   DBNAME     FILE_TYPE 	       MB	  GB	 #FILES
---------- ---------- ---------------- ---------- ---------- ----------
DATAC1	   DATABSE1   CONTROLFILE	     2316	   2	      1
		      DATAFILE		  7620756	7442	     49
		      DATAGUARDCONFIG	       16	   0	      2
		      ONLINELOG 	    82536	  81	     14
		      PARAMETERFILE		8	   0	      1
	   **********				  ----------
	   total_db					7525

	   DATABSE2   CONTROLFILE	      492	   0	      1
		      DATAFILE		  3081604	3009	     47
		      ONLINELOG 	      416	   0	      4
		      PARAMETERFILE	       16	   0	      2
		      PASSWORD			0	   0	      2
		      TEMPFILE		    83372	  81	      3
	   **********				  ----------
	   total_db					3090

	   DATABSE3   CONTROLFILE	      588	   1	      1
		      DATAFILE		  1430712	1397	      8
		      DATAGUARDCONFIG	       16	   0	      2
		      ONLINELOG 	   147816	 144	     18
		      PARAMETERFILE		8	   0	      1
	   **********				  ----------
	   total_db					1542
[...]

**********					  ----------
total_disk					       76868

SQL To list Orphan files per Database:

SET VERIFY OFF
SET LINESIZE 200
SET SERVEROUTPUT ON
SET PAGESIZE 50000

DECLARE
   cmd   CLOB;
BEGIN
   FOR c IN (SELECT name Diskgroup
               FROM V$ASM_DISKGROUP)
   LOOP
      FOR l
         IN (SELECT 'rm ' || files files
               FROM
                    (SELECT '+' || c.Diskgroup || files files, TYPE
                       FROM (    SELECT UPPER
                                        (
                                           SYS_CONNECT_BY_PATH (aa.name, '/')
                                        )
                                           files
                                      , aa.reference_index
                                      , b.TYPE
                                   FROM (SELECT file_number
                                              , alias_directory
                                              , name
                                              , reference_index
                                              , parent_index
                                           FROM v$asm_alias) aa
                                      , (SELECT parent_index
                                           FROM (SELECT parent_index
                                                   FROM v$asm_alias
                                                  WHERE     group_number =
                                                               (SELECT group_number
                                                                  FROM v$asm_diskgroup
                                                                 WHERE name =
                                                                          c.Diskgroup)
                                                        AND alias_index = 0)) a
                                      , (SELECT file_number, TYPE
                                           FROM (SELECT file_number, TYPE
                                                   FROM v$asm_file
                                                  WHERE group_number =
                                                           (SELECT group_number
                                                              FROM v$asm_diskgroup
                                                             WHERE name =
                                                                      c.Diskgroup)))
                                        b
                                  WHERE     aa.file_number = b.file_number(+)
                                        AND aa.alias_directory = 'N'
                                        AND b.TYPE IN
                                               ('DATAFILE'
                                              , 'ONLINELOG'
                                              , 'CONTROLFILE'
                                              , 'TEMPFILE')
                             START WITH aa.PARENT_INDEX = a.parent_index
                             CONNECT BY PRIOR aa.reference_index =
                                           aa.parent_index)
                      WHERE SUBSTR
                            (
                               files
                             , INSTR (files, '/', 1, 1)
                             ,   INSTR (files, '/', 1, 2)
                               - INSTR (files, '/', 1, 1)
                               + 1
                            ) =
                               (SELECT dbname
                                  FROM (SELECT    '/'
                                               || UPPER (db_unique_name)
                                               || '/'
                                                  dbname
                                          FROM v$database))
                     MINUS
                     (SELECT UPPER (name) files, 'DATAFILE' TYPE
                        FROM v$datafile
                      UNION ALL
                      SELECT UPPER (name) files, 'TEMPFILE' TYPE
                        FROM v$tempfile
                      UNION ALL
                      SELECT UPPER (name) files, 'CONTROLFILE' TYPE
                        FROM v$controlfile
                       WHERE name LIKE '+' || c.Diskgroup || '%'
                      UNION ALL
                      SELECT UPPER (name), 'CONTROLFILE' TYPE
                        FROM v$datafile_copy
                       WHERE deleted = 'NO'
                      UNION ALL
                      SELECT UPPER (MEMBER) files, 'ONLINELOG' TYPE
                        FROM v$logfile
                       WHERE MEMBER LIKE '+' || c.Diskgroup || '%')))
      LOOP
         DBMS_OUTPUT.put_line (l.files);
      END LOOP;
   END LOOP;
END;
/

Expected Output:

rm +DATA/XPTODB/CONTROLFILE/CURRENT.4928.955985765
rm +DATA/XPTODB/CONTROLFILE/CURRENT.4934.955986589
rm +DATA/XPTODB/CONTROLFILE/CURRENT.4962.955998825
rm +DATA/XPTODB/CONTROLFILE/CURRENT.5063.956480113
rm +DATA/XPTODB/CONTROLFILE/CURRENT.6374.955984145
rm +DATA/XPTODB/CONTROLFILE/CURRENT.7547.955968953
rm +DATA/XPTODB/DATAFILE/TBSEXEMPLE.4936.955985803
rm +DATA/XPTODB/DATAFILE/TBSEXEMPLE.4966.955998847
rm +DATA/XPTODB/DATAFILE/TBSEXEMPLE.7540.955968995
rm +DATA/XPTODB/DATAFILE/TBSEXEMPLE.7574.955984177
rm +DATA/XPTODB/DATAFILE/TBSEXEMPLE.4937.955985803
rm +DATA/XPTODB/DATAFILE/TBSEXEMPLE.4967.955998847
rm +DATA/XPTODB/DATAFILE/TBSEXEMPLE.7542.955968995
rm +DATA/XPTODB/DATAFILE/TBSEXEMPLE.7558.955984177
rm +DATA/XPTODB/DATAFILE/SYSAUX.4935.955986599
rm +DATA/XPTODB/DATAFILE/SYSAUX.4963.955998847
rm +DATA/XPTODB/DATAFILE/SYSAUX.6286.955984161
rm +DATA/XPTODB/DATAFILE/SYSAUX.7544.955968963
rm +DATA/XPTODB/DATAFILE/SYSTEM.4930.955986599
rm +DATA/XPTODB/DATAFILE/SYSTEM.4964.955998847
rm +DATA/XPTODB/DATAFILE/SYSTEM.7536.955968965

To run this for all databases on server (RAC Databases):

export ORAENV_ASK=NO
for DBSID in `ps -ef | grep ora_pmon | grep -v grep | awk -F_ '{ print $3}'` 
do
echo "######" ${DBSID}
export ORACLE_SID=${DBSID}
. oraenv
sqlplus / as sysdba
@script.sql
exit
done

Hope it helps you!

Amazon RDS: How to perform RMAN operations?

Hi all,

Need to performa any RMAN Operation from Amazon RDS? Maybe something like validating backups?

Well, we have some options under by using RDSADMIN_RMAN_UTIL package. See below an example for validate backup:

BEGIN
 rdsadmin.rdsadmin_rman_util.validate_database(
 p_validation_type => 'PHYSICAL+LOGICAL',
 p_parallel => 4,
 p_section_size_mb => 10,
 p_rman_to_dbms_output => FALSE);
END;
/

More info about it: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.Oracle.CommonDBATasks.RMAN.html

Hope it helps, cheers!

Checking Basic Licensing Info on a DB Server

Hi all!
So, I got a new client and started checking on his licensing and hardware. Then I realized how “non-standard” this is and Oracle should probably provide a better way to do it. So I decided to share a few things:

# Checking Oracle Version Installed:

[oracle@greporaSRV inventory]$ cd /opt/oraInventory/logs
[oracle@greporaSRV logs]$  grep "\- Database edition" installActions*.log
INFO: - Database edition : Standard Edition One (Create and configure a database)

# Checking number of Sockets

[root@greporaSRV ~]# cat /proc/cpuinfo | grep "physical id" | sort -u | wc -l
1

# Checking number of CPU Cores per Socket

[root@greporaSRV ~]# lscpu | grep 'socket'
Core(s) per socket:    4

Automatic Graceful Shutdown and Consistent Startup Method for Oracle DB on Linux

Hi all,

This is quite a common question whenever I arrive on any new company. The things is, there are more then one way to implement this depending on your environment, licensing and version.

So I decided to compile here some sort of summary for this:

1. Prefer to use Oracle Restart
This is the automated and validated method provided by Oracle, however it can be a bit confusing in some items which can lead us to think it’s not working. Here is a summary of the the configuration I recommend:

a) Configure database management to AUTOMATIC on SRVCTL

srvctl modify database -y AUTOMATIC
  • If AUTOMATIC (the default), the database is automatically restored to its previous running condition (started or stopped) upon restart of the database host computer.
  • If MANUAL, the database is never automatically restarted upon restart of the database host computer.

Refhttps://docs.oracle.com/cd/E11882_01/server.112/e25494/restart.htm#BABHHAHI

b) Set AUTO_START=always on CRSCTL

crsctl modify resource ora.grepora.db -attr AUTO_START=always
  • ALWAYS: Restarts the resource when the server restarts regardless of the state of the resource when the server stopped.
  • RESTORE: Restores the resource to the same state that it was in when the server stopped. Oracle Clusterware attempts to restart the resource if the value of TARGET was ONLINE before the server stopped.
  • NEVER: Oracle Clusterware never restarts the resource regardless of the state of the resource when the server stopped.

NOTE: On Oracle 11.2, the database auto start policy in the clusterware is restore, which means that clusterware will remember the last state of the database. As well as database, Oracle 11.2 comes by default with several important resources with attribute AUTO_START=restore in the profile.

NOTE2: 12c on you might need to use the flag “-unsupported” on command above (crsctl modify resource ora.grepora.db -attr AUTO_START=always – unsupported).

Refhttps://docs.oracle.com/cd/E11882_01/rac.112/e41959/resatt.htm#CHDFFEHJ

Observation: This is recommended for all the required components managed by those tools, like databases, asm, listener, diskgroups, etc.
I wrote an article about it with an script that I made by my own and can help you: https://grepora.com/2018/08/22/services-not-starting-automatically-with-crs-after-reboot/

A common problem: “I set the SRVCTL to Automatic, but databases still not starting automatically’.”
Explanation: When database Management policy is configured as AUTOMATIC and the resource of the database parameter AUTO_START is configured as restore, the cluster will restore its last state, because the cluster level is the first in the chain of commands, it overrides the database.

c) Save desired state of Pluggable Databases in case of Multitenant:
With the PDB in desired state, save it with command below:

ALTER PLUGGABLE DATABASE pdb1 SAVE STATE;

When the CDB start, it will bring the pdbs to it saved states.

2. As second Option, Oracle Provided Scripts

Oracle has some scripts to automate it in a standard and supported way. This is documented for 12.1 in Stopping and Starting Oracle Software.

NOTES:

  • Oracle 11gR2 documentation states the use of the dbstart and dbshut scripts are deprecated. It’s supposed to be replaced by Oracle Restart.
  • The Oracle 12c documentation has no mention of the deprecation of dbstart and dbshut and has reinstated the documentation about them (as I linked above). So, feel free to use dbstart and dbshut in a supported manner for all versions of the database.

I also wrote an article about those, with some info and scripts: https://grepora.com/2017/11/22/how-to-setup-automatic-startup-and-shutdown-of-an-oracle-database-on-linux-not-using-oracle-restart/

Observation: Item 1.c is still recommended here.

3. Community proven scripts

As a third option, we would have some community scripts, which are usually proven and doesn’t require us to remember or to code everything. I’d use some additional time reviewing and testing them though, as they are not Oracle provided/supported.

In general, I’d recommend the material produced by Tim Hall (Oracle Base): https://oracle-base.com/articles/linux/automating-database-startup-and-shutdown-on-linux

He has additional articles that may help for other versions:

  • Automating Shutdown and Startup (12.2)
  • Automating Shutdown and Startup (12.1)
  • Automating Shutdown and Startup (11.2)
  • Automating Shutdown and Startup (10.2)
  • Automating Startup and Shutdown (10.1)
  • Automating Database Startup and Shutdown (9.2)
  • Linux Services (systemd, systemctl)

 

Some Additional Twists:

  • The Oracle Restart configuration assume the CRS is left “enabled”. Disabling it means we don’t want it to start automatically. So, if you want the CRS to start with your server, it need to be enabled. After this, to start targets, depend on configurations as per mentioned on my previous post.
  •  Oracle will no execute any rpm change or relink automatically, as this is not part of any “restart” process. It may be required due any configuration change or corruption, and it cannot be automated.
  • Regarding gracefulness, it depends on the configuration you have on your SRVCTL too. It’s configured using stop and start option, as per example below:
srvctl modify database -d [db_unique_name] -s [start_options] -t [stop_options]
  • So for your case, it seems to me a complete command containing what was recommended on my previous post PLUS gracefulness, it would be:
srvctl modify database -d [db_unique_name] -s OPEN -t IMMEDIATE -y automatic

(Note the SRVCTL syntax can very on the versions. This one is valid for 11.2).

I hope this helps you on understanding the process.

See you next time!

ORA-00600: internal error code, arguments: [kghfrempty:ds]

Hi all,
Annoying thing, just because that sounded veeery unsual to me. I would expect a different error.

I have seen this “[kghfrempty:ds]” in past a couple times, and if you did see it too you may have noticed, this is almost as much generic as the ORA-600 itself. It happens because this is usually in the top of KGH errors in general. Some examples:

1.

kghnerror <- kghfrempty <- kghfrempty_ex <- qerhjFreeSpace
<- qerhjFetch <- qersoProcessULS <- qersoFetch <- opifch2 <- kpoal8
<- opiodr <- kpoodrc <- rpiswu2 <- kpoodr <- upirtrc
<- kpurcsc <- kpufch0 <- kpufch <- OCIStmtFetch2 <- qksanExecSql <- qksanAnalyzeSql

2.

kghfrempty <- kghfrempty_partial <- kghgex <- kghfnd <- kghalo <- kghgex <- kghalf <- ktmrProcessCRClone 
<- ktmbRead <- ktmrget_int <- ktmrget <- kdmsTransGet <- kdst_fetch_imc  <- kdsttgr  <- qertbFetch

And as being it can be related either to analyze bugs (as per stack 1), In-Memory Column Store – IMC (as per stack 2), create views, rollback over cluster, trigger creation, use function based indexes, or several other generic situations (there is no pattern, as you can see).

So what do to if you get this message? Well, generically, read all the MOS notes available and try to match you case. IF you match, pray to have a bug for your current version, or open an SR to either have a backport or start a new investigation.

A few examples of things to check:

  • ALERT: Bug 13605839 ORA-600 [ktbsdp1] ORA-600 [kghfrempty:ds] ORA-600 [kdBlkCheckError]. Corruption in Rollback with Clusterwide Global Transactions in RAC (Doc ID 1527740.1)
  • SQL Analyze Fails with ORA-600[kghfrempty:ds] (Doc ID 2247180.1)
    Bug 21556276 – dataguide createviewonpath hit ora 600 [kghfrempty:ds] (Doc ID 21556276.8)
  • Bug 20878625 – ORA-600[kghfrempty:ds] with IMC enabled (Doc ID 20878625.8)
  • Bug 19212166 – ORA-600 [KGHFREMPTY:DS] With Cache Line Alignment Enabled (Doc ID 19212166.8)
  • Bug 6797925 – ORA-600 [kghfrempty:ds] During Creation Of Trigger In XML Schema (Doc ID 6797925.8)
  • 11.1: ORA-600 [17147] AND [kghfrempty:ds] On Select Using Functional Indexes (Doc ID 884882.1)
  • ORA-600 [kghfrempty:ds] And ORA-600 [kghrcdepth:ds] While Gathering Statistics (Doc ID 1480132.1)

BUUT I just noticed some undocumented scenario causing this and it might help you on fixing this as well.
I noticed I was getting this error for different queries but always from same application. So I addressed a complete review on application layer and noticed the JDBC 7 on this application. Happens we had just upgraded from 12.1 to 19c.

SO, after upgrading jdbc, it started to work again properly and no MOS SR was required. So lesson learned: When you receive this kghfrempty:ds], speacially after migration, also check for your application JDBC version.

For compatibility Matrix: Starting With Oracle JDBC Drivers – Installation, Certification, and More! ( Doc ID 401934.1 )

Hope it helps, cheers!

ORA-65532: cannot alter or drop automatically created indexes

Hello all,
Quickly walking through this, as it was new to me (as for everybody, I guess): So what if I’m not happy with Oracle algorithm and I want to drop and AUTO Index on 19c?

Should be easy, like “drop index XXXX;” right? Wrong. See the struggle:

SQL> drop index GREPORA."SYS_AI_9xu652x5fyu5i";
drop index GREPORA."SYS_AI_9xu652x5fyu5i"
                 *
ERROR at line 1:
ORA-65532: cannot alter or drop automatically created indexes

OK, so let’s have a look on my auto created indexes. You can see them with flag AUTO as YES on query below:

SQL> select owner, index_name, auto, tablespace_name from dba_indexes natural where auto='YES';
OWNER              INDEX_NAME    AUTO    TABLESPACE_NAME
________ _______________________ _______ __________________
GREPORA  SYS_AI_9xu652x5fyu5i    YES     GREPORA
GREPORA  SYS_AI_few32swe423dw    YES     GREPORA
GREPORA  SYS_AI_94osd824n202f    YES     GREPORA

Ok, so I cannot drop, let me alter it and set it as unusable, then, as it would archive my intend anyway:

SQL> alter index ADMIN."SYS_AI_9xu652x5fyu5i" unusable;
alter index ADMIN."SYS_AI_9xu652x5fyu5i" unusable
 *
ERROR at line 1:
ORA-65532: cannot alter or drop automatically created indexes

Hmmm, so it means I cannot alter it as well?
Well, kind of, there is something I can do:

SQL> alter index ADMIN."SYS_AI_9xu652x5fyu5i" rebuild tablespace GREPORA2 online;
Index altered.

Noice, thanks for nothing. What is changed?
Well, you can use a workaround to move it to a new tablespace and then drop the tablespace:

SQL> drop tablespace GREPORA2 including contents;
Tablespace dropped.

And index will be gone:

SQL> select owner,index_name,auto,tablespace_name from dba_indexes natural where auto='YES';
OWNER              INDEX_NAME    AUTO    TABLESPACE_NAME
________ _______________________ _______ __________________
GREPORA  SYS_AI_few32swe423dw    YES     GREPORA
GREPORA  SYS_AI_94osd824n202f    YES     GREPORA

OK, so that’s bad, right… What the worse part? If you simply drop it, Oracle will probably recreate it, based on algorithm, right?
Gee, thanks for nothing (again)!

This, though, is easier to change: Simply alter the Audo Indexing to report mode:

exec dbms_auto_index.configure('AUTO_INDEX_MODE','REPORT');

This is interesting right?
Ok, we have more options, we can even exclude specific schemas from AUTO_INDEX, choose a default tablespace, define retentions and a lot of things.

As usual, the best reference I could find: https://oracle-base.com/articles/19c/automatic-indexing-19c

Just one thing you might want to consider and it’s not on Tim Hall’s post. Specially if the index is bad for one specific query, is to avoid using it during that specific query. We can do it with session/system parameter:

“_optimizer_use_auto_indexes”=OFF

Well, hope it helps you. Cheers!

Stop Migrating Databases to 12c!

If you are starting an upgrade plan, do it directly to 19c. Why?

Well, first let’s say that 19c is “equal” to 12.2.0.3 on previous version model, while 18c is 12.2.0.2, after latest 12.2.0.1. So, thinking reasonably, you would already plan for latest PSU on the future version, right? So this is 19c.

But not only this. It’s well known that Oracle’s Terminal Patchset is the Long Term Release for each version. And for 12.2 family this is 19c.

The image below (From Doc ID 742060.1) should help on clarifying this:

versions

In summary, since August 1st 2019:
– 11.2.0.4 is already on Extended Support (20% more expensive unless you have ULA)
– 12.1 family is already on Extended Support (20% more expensive unless you have ULA)

Which only leave us with 12.2 Family. Now I ask you, why jump to any “PSU” other than the final one?
Also, support on 12.2.0.1 goes only up to Nov 2020 and for 18c (12.2.0.2) only to Jun 2021, while 19c goes up to 2026 with ES/ULA!

I’m not even raising the great improvements we have on newer versions (if you are in 11.2, shame on you, you are basically 6 years behind the world technology wise).
Myself , I already migrated several databases on different clients to 19c and it’s running preeeetty fine. So trust on it!

Ok, so to migrate to 19c? (Some tips):
– To migrate directly to 19c you need to be on supported terminal patch set (11.2.0.4, 12.1.0.2) or on 12.2 family (12.2.0.1, 18c).
– Get in touch and use AutoUpgrade mechanism (you’ll love it!) – AutoUpgrade Tool (Doc ID 2485457.1)
– Check Upgrade Guide: https://docs.oracle.com/en/database/oracle/oracle-database/19/upgrd/index.html
– Have a look on Mike Dietrich blog for news: https://mikedietrichde.com/
– Test it before doing in prod.

Additional Reference:
Release Schedule of Current Database Releases (Doc ID 742060.1)
Very nice and complete post (in portuguese) on this topic from Ricardo Portilho.

I like very much the table below as well:

NewScreenshot 2019-08-13 às 15.11.34

See you!

Quick ways to transfer files to Oracle Support – MOS directly from the Database Server

Hi all,

This will be a quick one but helped me a lot last week.

I was working on an Ora-0600 issue on a 2 node RAC cluster. Working with Oracle Support I was asked to transfer a TFA data as well ADRCI data so they could move forward with the SR, very common process right?

After I generated all the files, it was over 3gb of data to be sent over. If I was to get the file from the database server to my laptop I would need to transfer these files multiple times due to the jump servers in the middle.

This time I was luck as the database server was able to ping the site https://transport.oracle.com so I tried sending the files directly to MOS.

I will show you how I did it. This is not a mistery but can really save you time

You have 2 options which are straight forward

Using curl

curl -T FILE_YOU_WANT_TO_SEND -u MOS_USER https://transport.oracle.com/upload/issue/SR_NUMBER/

Using tfacfl

tfactl upload -sr SR_NUMBER -user MOS_USER FILE_YOU_WANT_TO_SEND

On both you need to provide your MOS credentials the SR number and the file you want o upload

After the upload is done, you will see in the attachments in the SR that the files is with the status like “Transferring to the SR” after a few minutes later you will see the update on the SR saying that the file was uploaded to the SR.

Hope this helps!

Elisson Almeida

Invalid system views after upgrading from 12c to 18c – X_$KGLOB and X_$KGLCURSOR

I was working on a client upgrade from 12c to 18c. The upgrade was done manually. After the dbupgrade 2 system views got invalid, X_$KGLCURSOR and X_$KGLOB

OWNER OBJECT_NAME OBJECT_TYPE STATUS
------------ -------------------------------------------------- ---------------------------------------- -------
SYS X_$KGLOB VIEW INVALID
SYS X_$KGLCURSOR VIEW INVALID

Man system objects invalid after an upgrade is never good.

Troubleshooting, I got the view text and tried to execute its sql

SQL> select TEXT from dba_Views where view_name = 'X_$KGLOB';

TEXT
------------------------------------------------------------------------------------------------------------------------------------------------------
SELECT "ADDR","INDX","INST_ID","CON_ID","KGLHDADR","KGLHDPAR","KGLHDCLT","KGLNAOWN","KGLNAOBJ","KGLFNOBJ","KGLNADLK","KGLNAHSH","KGLNAHSV","KGLNATIM",
"KGLNAPTM","KGLHDNSP","KGLHDNSD","KGLHDLMD","KGLHDPMD","KGLHDFLG","KGLHDOBJ","KGLHDLDC","KGLHDIVC","KGLHDEXC","KGLHDLKC","KGLHDKMK","KGLHDDMK","KGLHDA
MK","KGLOBFLG","KGLOBSTA","KGLOBTYP","KGLOBTYD","KGLOBHMK","KGLOBHS0","KGLOBHS1","KGLOBHS2","KGLOBHS3","KGLOBHS4","KGLOBHS5","KGLOBHS6","KGLOBHS7","KG
LOBHD0","KGLOBHD1","KGLOBHD2","KGLOBHD3","KGLOBHD4","KGLOBHD5","KGLOBHD6","KGLOBHD7","KGLOBPC0","KGLOBPC6","KGLOBTP0","KGLOBT00","KGLOBT01","KGLOBT02"
,"KGLOBT03","KGLOBT04","KGLOBT05","KGLOBT35","KGLOBT06","KGLOBT07","KGLOBT08","KGLOBT09","KGLOBT10","KGLOBT11","KGLOBT12","KGLOBT13","KGLOBT14","KGLOB
T15","KGLOBT16","KGLOBT17","KGLOBT18","KGLOBT19","KGLOBT20","KGLOBT21","KGLOBT22","KGLOBT23","KGLOBT24","KGLOBT25","KGLOBT26","KGLOBT28","KGLOBT29","K
GLOBT30","KGLOBT31","KGLOBT27","KGLOBT32","KGLOBT33","KGLOBWAP","KGLOBWCC","KGLOBWCL","KGLOBWUI","KGLOBWDW","KGLOBT42","KGLOBT43","KGLOBT44","KGLOBT45
","KGLOBT46","KGLOBT47","KGLOBT49","KGLOBT50","KGLOBT52","KGLOBT53","KGLOBTL0","KGLOBTL1","KGLOBTS0","KGLOBTS1","KGLOBTN0","KGLOBTN1","KGLOBTN2","KGLO
BTN3","KGLOBTN4","KGLOBTN5","KGLOBTS2","KGLOBTS3","KGLOBTS5","KGLOBTT0","KGLOBCCE","KGLOBCCEH","KGLOBCLA","KGLOBCLC","KGLOBCCC","KGLOBTS4","KGLOBCBCA"
,"KGLOBT48","KGLOBDSO","KGLOBDEX","KGLOBDPX","KGLOBDLD","KGLOBDIV","KGLOBDPS","KGLOBDDR","KGLOBDDW","KGLOBDBF","KGLOBDRO","KGLOBDCP","KGLOBDEL","KGLOB
DFT","KGLOBDEF","KGLOBDUI","KGLOBDCL","KGLOBDAP","KGLOBDCC","KGLOBDPL","KGLOBDJV","KGLOBACS","KGLOBTS6","KGLOBTS7","KGLOBT54","KGLOBT55","KGLOBT56","K
GLOBT57","KGLOBDCO","KGLOBDCI","KGLOBDRR","KGLOBDRB","KGLOBDWR","KGLOBDWB","KGLOBT58","KGLOBDOR","KGLHDMTX","KGLHDMVL","KGLHDMSP","KGLHDMGT","KGLHDDMT
X","KGLHDDVL","KGLHDDSP","KGLHDDGT","KGLHDBID","KGLHDBMTX","KGLHDBVL","KGLHDBSP","KGLHDBGT","KGLOBT59","KGLOBDCU","KGLOBPROP","KGLNACON","KGLNAO1N","K
GLHDMAR","KGLMPFLG","KGLOBAQP","KGLOBT60","KGLOBT61","KGLIMSCANS","KGLIMBYTESUNCOMP","KGLIMBYTESINMEM" FROM X$KGLOB


1 row selected.
SQL> SELECT "ADDR","INDX","INST_ID","CON_ID","KGLHDADR","KGLHDPAR","KGLHDCLT","KGLNAOWN","KGLNAOBJ","KGLFNOBJ","KGLNADLK","KGLNAHSH","KGLNAHSV","KGLNATIM","KGLNAPTM","KGLHDNSP","KGLHDNSD","KGLHDLMD","KGLHDPMD","KGLHDFLG","KGLHDOBJ","KGLHDLDC","KGLHDIVC","KGLHDEXC","KGLHDLKC","KGLHDKMK","KGLHDDMK","KGLHDAMK","KGLOBFLG","KGLOBSTA","KGLOBTYP","KGLOBTYD","KGLOBHMK","KGLOBHS0","KGLOBHS1","KGLOBHS2","KGLOBHS3","KGLOBHS4","KGLOBHS5","KGLOBHS6","KGLOBHS7","KGLOBHD0","KGLOBHD1","KGLOBHD2","KGLOBHD3","KGLOBHD4","KGLOBHD5","KGLOBHD6","KGLOBHD7","KGLOBPC0","KGLOBPC6","KGLOBTP0","KGLOBT00","KGLOBT01","KGLOBT02","KGLOBT03","KGLOBT04","KGLOBT05","KGLOBT35","KGLOBT06","KGLOBT07","KGLOBT08","KGLOBT09","KGLOBT10","KGLOBT11","KGLOBT12","KGLOBT13","KGLOBT14","KGLOBT15","KGLOBT16","KGLOBT17","KGLOBT18","KGLOBT19","KGLOBT20","KGLOBT21","KGLOBT22","KGLOBT23","KGLOBT24","KGLOBT25","KGLOBT26","KGLOBT28","KGLOBT29","KGLOBT30","KGLOBT31","KGLOBT27","KGLOBT32","KGLOBT33","KGLOBWAP","KGLOBWCC","KGLOBWCL","KGLOBWUI","KGLOBWDW","KGLOBT42","KGLOBT43","KGLOBT44","KGLOBT45","KGLOBT46","KGLOBT47","KGLOBT49","KGLOBT50","KGLOBT52","KGLOBT53","KGLOBTL0","KGLOBTL1","KGLOBTS0","KGLOBTS1","KGLOBTN0","KGLOBTN1","KGLOBTN2","KGLOBTN3","KGLOBTN4","KGLOBTN5","KGLOBTS2","KGLOBTS3","KGLOBTS5","KGLOBTT0","KGLOBCCE","KGLOBCCEH","KGLOBCLA","KGLOBCLC","KGLOBCCC","KGLOBTS4","KGLOBCBCA","KGLOBT48","KGLOBDSO","KGLOBDEX","KGLOBDPX","KGLOBDLD","KGLOBDIV","KGLOBDPS","KGLOBDDR","KGLOBDDW","KGLOBDBF","KGLOBDRO","KGLOBDCP","KGLOBDEL","KGLOBDFT","KGLOBDEF","KGLOBDUI","KGLOBDCL","KGLOBDAP","KGLOBDCC","KGLOBDPL","KGLOBDJV","KGLOBACS","KGLOBTS6","KGLOBTS7","KGLOBT54","KGLOBT55","KGLOBT56","KGLOBT57","KGLOBDCO","KGLOBDCI","KGLOBDRR","KGLOBDRB","KGLOBDWR","KGLOBDWB","KGLOBT58","KGLOBDOR","KGLHDMTX","KGLHDMVL","KGLHDMSP","KGLHDMGT","KGLHDDMTX","KGLHDDVL","KGLHDDSP","KGLHDDGT","KGLHDBID","KGLHDBMTX","KGLHDBVL","KGLHDBSP","KGLHDBGT","KGLOBT59","KGLOBDCU","KGLOBPROP","KGLNACON","KGLNAO1N","KGLHDMAR","KGLMPFLG","KGLOBAQP","KGLOBT60","KGLOBT61","KGLIMSCANS","KGLIMBYTESUNCOMP","KGLIMBYTESINMEM" FROM X$KGLOB;
SELECT "ADDR","INDX","INST_ID","CON_ID","KGLHDADR","KGLHDPAR","KGLHDCLT","KGLNAOWN","KGLNAOBJ","KGLFNOBJ","KGLNADLK","KGLNAHSH","KGLNAHSV","KGLNATIM","KGLNAPTM","KGLHDNSP","KGLHDNSD","KGLHDLMD","KGLHDPMD","KGLHDFLG","KGLHDOBJ","KGLHDLDC","KGLHDIVC","KGLHDEXC","KGLHDLKC","KGLHDKMK","KGLHDDMK","KGLHDAMK","KGLOBFLG","KGLOBSTA","KGLOBTYP","KGLOBTYD","KGLOBHMK","KGLOBHS0","KGLOBHS1","KGLOBHS2","KGLOBHS3","KGLOBHS4","KGLOBHS5","KGLOBHS6","KGLOBHS7","KGLOBHD0","KGLOBHD1","KGLOBHD2","KGLOBHD3","KGLOBHD4","KGLOBHD5","KGLOBHD6","KGLOBHD7","KGLOBPC0","KGLOBPC6","KGLOBTP0","KGLOBT00","KGLOBT01","KGLOBT02","KGLOBT03","KGLOBT04","KGLOBT05","KGLOBT35","KGLOBT06","KGLOBT07","KGLOBT08","KGLOBT09","KGLOBT10","KGLOBT11","KGLOBT12","KGLOBT13","KGLOBT14","KGLOBT15","KGLOBT16","KGLOBT17","KGLOBT18","KGLOBT19","KGLOBT20","KGLOBT21","KGLOBT22","KGLOBT23","KGLOBT24","KGLOBT25","KGLOBT26","KGLOBT28","KGLOBT29","KGLOBT30","KGLOBT31","KGLOBT27","KGLOBT32","KGLOBT33","KGLOBWAP","KGLOBWCC","KGLOBWCL","KGLOBWUI","KGLOBWDW","KGLOBT42","KGLOBT43","KGLOBT44","KGLOBT45","KGLOBT46","KGLOBT47","KGLOBT49","KGLOBT50","KGLOBT52","KGLOBT53","KGLOBTL0","KGLOBTL1","KGLOBTS0","KGLOBTS1","KGLOBTN0","KGLOBTN1","KGLOBTN2","KGLOBTN3","KGLOBTN4","KGLOBTN5","KGLOBTS2","KGLOBTS3","KGLOBTS5","KGLOBTT0","KGLOBCCE","KGLOBCCEH","KGLOBCLA","KGLOBCLC","KGLOBCCC","KGLOBTS4","KGLOBCBCA","KGLOBT48","KGLOBDSO","KGLOBDEX","KGLOBDPX","KGLOBDLD","KGLOBDIV","KGLOBDPS","KGLOBDDR","KGLOBDDW","KGLOBDBF","KGLOBDRO","KGLOBDCP","KGLOBDEL","KGLOBDFT","KGLOBDEF","KGLOBDUI","KGLOBDCL","KGLOBDAP","KGLOBDCC","KGLOBDPL","KGLOBDJV","KGLOBACS","KGLOBTS6","KGLOBTS7","KGLOBT54","KGLOBT55","KGLOBT56","KGLOBT57","KGLOBDCO","KGLOBDCI","KGLOBDRR","KGLOBDRB","KGLOBDWR","KGLOBDWB","KGLOBT58","KGLOBDOR","KGLHDMTX","KGLHDMVL","KGLHDMSP","KGLHDMGT","KGLHDDMTX","KGLHDDVL","KGLHDDSP","KGLHDDGT","KGLHDBID","KGLHDBMTX","KGLHDBVL","KGLHDBSP","KGLHDBGT","KGLOBT59","KGLOBDCU","KGLOBPROP","KGLNACON","KGLNAO1N","KGLHDMAR","KGLMPFLG","KGLOBAQP","KGLOBT60","KGLOBT61","KGLIMSCANS","KGLIMBYTESUNCOMP","KGLIMBYTESINMEM" FROM X$KGLOB
*
ERROR at line 1:
ORA-00904: "KGLOBT24": invalid identifier
SQL> select TEXT from dba_Views where view_name = 'X_$KGLCURSOR'

TEXT
------------------------------------------------------------------------------------------------------------------------------------------------------
SELECT "ADDR","INDX","INST_ID","CON_ID","KGLHDADR","KGLHDPAR","KGLHDCLT","KGLNAOWN","KGLNAOBJ","KGLFNOBJ","KGLNADLK","KGLNAHSH","KGLNAHSV","KGLNATIM",
"KGLNAPTM","KGLHDNSP","KGLHDNSD","KGLHDLMD","KGLHDPMD","KGLHDFLG","KGLHDOBJ","KGLHDLDC","KGLHDIVC","KGLHDEXC","KGLHDLKC","KGLHDKMK","KGLHDDMK","KGLHDA
MK","KGLOBFLG","KGLOBSTA","KGLOBTYP","KGLOBTYD","KGLOBHMK","KGLOBHS0","KGLOBHS1","KGLOBHS2","KGLOBHS3","KGLOBHS4","KGLOBHS5","KGLOBHS6","KGLOBHS7","KG
LOBHD0","KGLOBHD1","KGLOBHD2","KGLOBHD3","KGLOBHD4","KGLOBHD5","KGLOBHD6","KGLOBHD7","KGLOBPC0","KGLOBPC6","KGLOBTP0","KGLOBT00","KGLOBT01","KGLOBT02"
,"KGLOBT03","KGLOBT04","KGLOBT05","KGLOBT35","KGLOBT06","KGLOBT07","KGLOBT08","KGLOBT09","KGLOBT10","KGLOBT11","KGLOBT12","KGLOBT13","KGLOBT14","KGLOB
T15","KGLOBT16","KGLOBT17","KGLOBT18","KGLOBT19","KGLOBT20","KGLOBT21","KGLOBT22","KGLOBT23","KGLOBT24","KGLOBT25","KGLOBT26","KGLOBT28","KGLOBT29","K
GLOBT30","KGLOBT31","KGLOBT27","KGLOBT32","KGLOBT33","KGLOBWAP","KGLOBWCC","KGLOBWCL","KGLOBWUI","KGLOBWDW","KGLOBT42","KGLOBT43","KGLOBT44","KGLOBT45
","KGLOBT46","KGLOBT47","KGLOBT49","KGLOBT50","KGLOBT52","KGLOBT53","KGLOBTL0","KGLOBTL1","KGLOBTS0","KGLOBTS1","KGLOBTN0","KGLOBTN1","KGLOBTN2","KGLO
BTN3","KGLOBTN4","KGLOBTN5","KGLOBTS2","KGLOBTS3","KGLOBTS5","KGLOBTT0","KGLOBCCE","KGLOBCCEH","KGLOBCLA","KGLOBCLC","KGLOBCCC","KGLOBTS4","KGLOBCBCA"
,"KGLOBT48","KGLOBDSO","KGLOBDEX","KGLOBDPX","KGLOBDLD","KGLOBDIV","KGLOBDPS","KGLOBDDR","KGLOBDDW","KGLOBDBF","KGLOBDRO","KGLOBDCP","KGLOBDEL","KGLOB
DFT","KGLOBDEF","KGLOBDUI","KGLOBDCL","KGLOBDAP","KGLOBDCC","KGLOBDPL","KGLOBDJV","KGLOBACS","KGLOBTS6","KGLOBTS7","KGLOBT54","KGLOBT55","KGLOBT56","K
GLOBT57","KGLOBDCO","KGLOBDCI","KGLOBDRR","KGLOBDRB","KGLOBDWR","KGLOBDWB","KGLOBT58","KGLOBDOR","KGLHDMTX","KGLHDMVL","KGLHDMSP","KGLHDMGT","KGLHDDMT
X","KGLHDDVL","KGLHDDSP","KGLHDDGT","KGLHDBID","KGLHDBMTX","KGLHDBVL","KGLHDBSP","KGLHDBGT","KGLOBT59","KGLOBDCU","KGLOBPROP","KGLNACON","KGLNAO1N","K
GLHDMAR","KGLMPFLG","KGLOBAQP","KGLOBT60","KGLOBT61","KGLIMSCANS","KGLIMBYTESUNCOMP","KGLIMBYTESINMEM" FROM X$KGLCURSOR


1 row selected.
SQL> SELECT "ADDR","INDX","INST_ID","CON_ID","KGLHDADR","KGLHDPAR","KGLHDCLT","KGLNAOWN","KGLNAOBJ","KGLFNOBJ","KGLNADLK","KGLNAHSH","KGLNAHSV","KGLNATIM",
"KGLNAPTM","KGLHDNSP","KGLHDNSD","KGLHDLMD","KGLHDPMD","KGLHDFLG","KGLHDOBJ","KGLHDLDC","KGLHDIVC","KGLHDEXC","KGLHDLKC","KGLHDKMK","KGLHDDMK","KGLHDA
MK","KGLOBFLG","KGLOBSTA","KGLOBTYP","KGLOBTYD","KGLOBHMK","KGLOBHS0","KGLOBHS1","KGLOBHS2","KGLOBHS3","KGLOBHS4","KGLOBHS5","KGLOBHS6","KGLOBHS7","KG
LOBHD0","KGLOBHD1","KGLOBHD2","KGLOBHD3","KGLOBHD4","KGLOBHD5","KGLOBHD6","KGLOBHD7","KGLOBPC0","KGLOBPC6","KGLOBTP0","KGLOBT00","KGLOBT01","KGLOBT02"
,"KGLOBT03","KGLOBT04","KGLOBT05","KGLOBT35","KGLOBT06","KGLOBT07","KGLOBT08","KGLOBT09","KGLOBT10","KGLOBT11","KGLOBT12","KGLOBT13","KGLOBT14","KGLOB
T15","KGLOBT16","KGLOBT17","KGLOBT18","KGLOBT19","KGLOBT20","KGLOBT21","KGLOBT22","KGLOBT23","KGLOBT24","KGLOBT25","KGLOBT26","KGLOBT28","KGLOBT29","K
GLOBT30","KGLOBT31","KGLOBT27","KGLOBT32","KGLOBT33","KGLOBWAP","KGLOBWCC","KGLOBWCL","KGLOBWUI","KGLOBWDW","KGLOBT42","KGLOBT43","KGLOBT44","KGLOBT45
","KGLOBT46","KGLOBT47","KGLOBT49","KGLOBT50","KGLOBT52","KGLOBT53","KGLOBTL0","KGLOBTL1","KGLOBTS0","KGLOBTS1","KGLOBTN0","KGLOBTN1","KGLOBTN2","KGLO
BTN3","KGLOBTN4","KGLOBTN5","KGLOBTS2","KGLOBTS3","KGLOBTS5","KGLOBTT0","KGLOBCCE","KGLOBCCEH","KGLOBCLA","KGLOBCLC","KGLOBCCC","KGLOBTS4","KGLOBCBCA"
,"KGLOBT48","KGLOBDSO","KGLOBDEX","KGLOBDPX","KGLOBDLD","KGLOBDIV","KGLOBDPS","KGLOBDDR","KGLOBDDW","KGLOBDBF","KGLOBDRO","KGLOBDCP","KGLOBDEL","KGLOB
DFT","KGLOBDEF","KGLOBDUI","KGLOBDCL","KGLOBDAP","KGLOBDCC","KGLOBDPL","KGLOBDJV","KGLOBACS","KGLOBTS6","KGLOBTS7","KGLOBT54","KGLOBT55","KGLOBT56","K
GLOBT57","KGLOBDCO","KGLOBDCI","KGLOBDRR","KGLOBDRB","KGLOBDWR","KGLOBDWB","KGLOBT58","KGLOBDOR","KGLHDMTX","KGLHDMVL","KGLHDMSP","KGLHDMGT","KGLHDDMT
X","KGLHDDVL","KGLHDDSP","KGLHDDGT","KGLHDBID","KGLHDBMTX","KGLHDBVL","KGLHDBSP","KGLHDBGT","KGLOBT59","KGLOBDCU","KGLOBPROP","KGLNACON","KGLNAO1N","K
GLHDMAR","KGLMPFLG","KGLOBAQP","KGLOBT60","KGLOBT61","KGLIMSCANS","KGLIMBYTESUNCOMP","KGLIMBYTESINMEM" FROM X$KGLCURSOR
2 3 4 5 6 7 8 9 10 11 12 13 14 15 
SQL> /
X","KGLHDDVL","KGLHDDSP","KGLHDDGT","KGLHDBID","KGLHDBMTX","KGLHDBVL","KGLHDBSP","KGLHDBGT","KGLOBT59","KGLOBDCU","KGLOBPROP","KGLNACON","KGLNAO1N","K
*
ERROR at line 13:
ORA-00904: "KGLHDMAR": invalid identifier

Describing the source tables used by the views, the columns that generated the error do not exist there. What?!

Looking up MOS – found this note, Invalid x_$ Objects After Upgrade (Doc ID 361757.1)

Although this note does not mention these views explicitly there is a part of that says that

"This is NOT a dictionary corruption, but the result of an unsupported operation.

.....

These x_$ views do not compile because they are based on x$ tables which have changed (columns dropped) and/or do not exist (X$ table dropped).

New (default) 10g & 11g databases do not have views with the prefix x_$.

Please note that this solution can be applied to any and all x_$ views found invalid after upgrade/migration.
These tables are built on x$ tables in the previous release and the same rules still apply."

So these are save to ignore or drop them.

Hope this helps.

Elisson Almeida

Waiting for ASM to startup after upgrading to Oracle 18c

Hi,
Something strange happened to me and I would like to share with you. I was upgrading one of my Virtual Machine which was an Oracle Restart – SIHA,  running 11.2.0.4 to 18c on Oracle Linux 6.9.

The upgrade went fine using the GUI no issues nor strange log messages, but the issue happened when I rebooted the VM. For some reason (which I don’t knew) the startup was hanging when trying to start HAS and then it outputted the error below:

PRCR-1070 :Failed to check if resource ora.asm is registered
CRS-0184 : Cannot communicate with the CRS daemon
Waiting for ASM to startup

Looking up on MOS I found this note – Oracle Linux 6 server hangs on boot with error: Waiting for ASM to startup (Doc ID 2495023.1)

The startup was hanging waiting on acfssihamount but as ASM was not up yet and it could not start.

To fix it you need to boot OS in rescue mode and execute

chroot /mnt/sysimage

Once you do that you will be able to change the startup configuration settings using chkconfig and disable acfssihamount

chkconfig --list acfssihamount
chkconfig acfssihamount off
chkconfig --list acfssihamount

After this exit out the rescue mode and the system should boot normally.

Hope it helps!

Elisson Almeida