CRS Not Starting after Removing OS User: How to Workaround and How to Solve!

Hello all!
Turns that a few days ago a client reached me because his CRSD was simply not starting. Like this:

[root@proddb proddb]$ ./crsctl start res ora.crsd -init
CRS-2672: Attempting to start 'ora.crsd' on 'proddb'
CRS-2676: Start of 'ora.crsd' on 'proddb' succeeded

[root@proddb proddb]$ ps -ef |grep crsd
root 19217 13424 0 11:53 pts/0 00:00:00 grep crsd

After some investigation, I found the following:

2017-01-24 14:00:06.859: [ CRSSEC][1690195712]{1:51052:2} Exception: OwnerEntry construction failed to retrieve user id by name with ACL string: owner:jacknobody:rwx and error: 1
2017-01-24 14:00:06.912: [ CRSSEC][1690195712]{1:51052:2} Exception: ACL entry creation failed for: owner:jacknobody:rwx

Hmmm, seems some CRS resources are owned by “Jack Nobody”… Turns that I this us was removed from OS:

[root@proddb proddb]$ cat /etc/passwd |grep jacknobody
[root@proddb proddb]$ 

What to do now?

More“CRS Not Starting after Removing OS User: How to Workaround and How to Solve!”

EMagent 12c with high number of threads causing “su: cannot set user id: Resource temporarily unavailable”

Hello all,
Yeah, these days I got some errors and when validating the server found the fllowing error:

su: cannot set user id: Resource temporarily unavailable

As you can imagine, in order to fix the issue, I adjusted the /etc/security/limits.conf increasing oracle nprocs to:

oracle           soft    nproc           4047
oracle           hard    nproc           20384

Ok, turns out that after a while I got the same errors again…
After some investigating I find that the EM Agent process was with 5020 threads!
Take a look:

More“EMagent 12c with high number of threads causing “su: cannot set user id: Resource temporarily unavailable””

How To Delete Archive Logs From Downstream Database

After deploy GoldenGate with downstream database option archives from downstream database not cleaning ? All database transaction are gracefull, but archives (from downstream) are not cleaning?

Are no log transactions registered on GG Extract?
check this:

GGSCI (dbcloud) 3> send ETL01 showtrans

Sending SHOWTRANS request to EXTRACT ETL01 ...

Oldest redo log files necessary to restart Extract are:
------------------------------------------------------------
XID:                  14.6.3931092
Items:                1
Extract:              ETL01
Redo Thread:          1
Start Time:           2017-00-00:00:00:00
SCN:                  1682.4049305132 (7228184297004)
Redo Seq:             8612
Redo RBA:             20965491728
Status:               Running
------------------------------------------------------------

If you on same case, make this:

It should reach (and delete) JUST archived log that have already read by GoldenGate.

set serveroutput on size unlimited
set line 1000
set trimsp on
set feed off
set pages 5000
set pagesize 0
spool [[ some_dir ]]/delete_archives__dowstream_goldengate.sh

-- SR 3-14409179111 - Golden Gate Configuration, How To Delete Archive Logs From Downstream Database (Doc ID 2011174.1)
SELECT 'rm ' || r.NAME
-- case when (r.next_scn > c.required_checkpoint_scn) then 'NO' else 'YES' end purgable
FROM DBA_REGISTERED_ARCHIVED_LOG r, DBA_CAPTURE c
WHERE r.CONSUMER_NAME = c.CAPTURE_NAME and r.source_database = c.source_database and r.next_scn < ( select min(required_checkpoint_scn) from dba_capture where captur
e_type='DOWNSTREAM')
and purgeable='YES'
order by modified_time;
spool off

Schedule the output in automation tool (crontab / dba_job / windows job scheduler, (whatever)):

GOLDENGATE process show RUNNING status, when not RUNNING.

This might occur in case of crash on host or database node where goldengate is running.

Goldengate maintains information about the process status in the file system, when the server crashes the file that has PID data is not released, and the process status in GGSCI shows erroneous information.

In GGSCI, process status it’s OK:

GGSCI (greporaLAB.net) 7> info mgr

Manager is running (IP port greporaLAB.net.7809, Process ID 42064).

If you try to stop process, you will not be able to:

More“GOLDENGATE process show RUNNING status, when not RUNNING.”

Shellscript: Which database is this service_name from?

Hey!
Strange question, right? You should know you applications and databases and how everything works, right?

Happen that some days ago a client requested me to identify from which database is a service_name. Easy? Not if the server have 46 database instances running…

Ok, now how can I can this information as quick as possible?

I know that the command below give me all distinct databases in CRS:

/u01/app/12.1.0.2/grid/bin/crsctl stat res -t -w "TYPE = ora.database.type"

Now, lets make some sed magic:

[root@nprd01dbadm01 ~]# /u01/app/12.1.0.2/grid/bin/crsctl stat res -t -w "TYPE = ora.database.type" |grep ora. |sed 's/ora./srvctl status service -d /' |sed 's/.db/ |grep \$1/' | head -1
srvctl status service -d dba |grep $1

More“Shellscript: Which database is this service_name from?”

GoldenGate Integrated Capture and Integrated Replicat Healthcheck Script

GoldenGate integrated Extract gives to dbas powerful tool to check GoldenGate’s operation in database, this package can  be found to download on Doc ID 1448324.1.

This Healthcheck is similar AWR reports and it been very useful to find some error or bottleneck.

Tool give some advices and parameter tips.

Let’s check my lab HC topics

Environment overview:lab1

Performance tips:

tip-2

tip-1

This HC uses system views created by OGG, so you can customize you own HC 🙂

Maiquel.

Backup Not Backuped Archivelogs and Delete Input

Hi all!
Sometimes you are caught in a situation where your database is not backuping archivelogs and need to generate a quick backup commands for those are not backuped yet and deleting it, right?
I saw this situation in this archived discussion at OTN. Unfortunately I couldn’t give my answer… But it’s how I do:

select
'backup archivelog from sequence '||sequence#||' until sequence '||sequence#||' thread '||thread#||' filesperset=1 delete input;',first_time
from v$archived_log where backup_count=0 and name is not null
order by first_time desc;

It generates an output like:

greporadb> select
  2  'backup archivelog from sequence '||sequence#||' until sequence '||sequence#||' thread '||thread#||' filesperset=1 delete input;',first_time
  3  from v$archived_log where backup_count=0 and name is not null
  4  order by first_time desc;
'BACKUPARCHIVELOGFROMSEQUEN                                                                                                                                             
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
backup archivelog from sequence 152153 until sequence 152153 thread 1 filesperset=1 delete input;                                                                       
backup archivelog from sequence 152152 until sequence 152152 thread 1 filesperset=1 delete input;                                                                       
backup archivelog from sequence 152151 until sequence 152151 thread 1 filesperset=1 delete input;                                                                       
3 rows selected.

And be happy!

But an observation! It not works this way for databases with dataguard. For these cases you’ll need to add “and name’&dgname’” at select where clause… 😉

See ya!
Matheus.

ORA-01153: an incompatible media recovery is active

When trying to start or increase parallel of recover manager on datagauard (MRP):

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT FROM SESSION;
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT FROM SESSION;
*
ERROR at line 1:
ORA-01153: an incompatible media recovery is active

I simply happen because there already are a process runnning, let’s check:

More“ORA-01153: an incompatible media recovery is active”

Saving database space with ASSM

It’s good way reclaim WASTED space from tables and index using  the Segment Advisor.

To perform an database reclaim procedure using Automatic Segment Space Management (ASSM) it is preferred to create tablespaces with below option:

grepdb> CREATE TABLESPACE HR
DATAFILE '+GREPORADG/'
SIZE 10M EXTENT MANAGEMENT LOCAL
SEGMENT SPACE MANAGEMENT AUTO;

Only tablespaces with segment space auto are eligible to Segment Advisor.

To manually run the Segment Advisor on OEM:

guid-65f07e4f-0482-47df-bdf9-8d34b625093a-default

It will save some database storage area, and make it more effective cause by LHWM/HHWM.

Maiquel.

After adding Datafile in Primary the MRP Stopped in Physical Standby (Dataguard)

Hi all!
After add a datafile in PRIMARY database, the STANDBY MRP stopped. An “ALTER DATABASE RECOVER MANAGED STANDBY DATABASE” does not solved te problem, as you see:

SQL> SELECT SEQUENCE#, Name, APPLIED FROM V$ARCHIVED_LOG where APPLIED  'YES' and SEQUENCE#  ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION;
Database altered.
SQL> SELECT SEQUENCE#, Name, APPLIED FROM V$ARCHIVED_LOG where APPLIED  'YES' and SEQUENCE# < (select max(SEQUENCE#) -1 from V$ARCHIVED_LOG);
SEQUENCE#  NAME                                                                              APPLIED
----------  --------------------------------------------------------------------------------  ---------
15075  /db/u1004/oracle/admin/MYDB/arch/arch_1_823102978_15075.arc                    NO

More“After adding Datafile in Primary the MRP Stopped in Physical Standby (Dataguard)”