CRS Not Starting after Removing OS User: How to Workaround and How to Solve!

Hello all!
Turns that a few days ago a client reached me because his CRSD was simply not starting. Like this:

[root@proddb proddb]$ ./crsctl start res ora.crsd -init
CRS-2672: Attempting to start 'ora.crsd' on 'proddb'
CRS-2676: Start of 'ora.crsd' on 'proddb' succeeded

[root@proddb proddb]$ ps -ef |grep crsd
root 19217 13424 0 11:53 pts/0 00:00:00 grep crsd

After some investigation, I found the following:

2017-01-24 14:00:06.859: [ CRSSEC][1690195712]{1:51052:2} Exception: OwnerEntry construction failed to retrieve user id by name with ACL string: owner:jacknobody:rwx and error: 1
2017-01-24 14:00:06.912: [ CRSSEC][1690195712]{1:51052:2} Exception: ACL entry creation failed for: owner:jacknobody:rwx

Hmmm, seems some CRS resources are owned by “Jack Nobody”… Turns that I this us was removed from OS:

[root@proddb proddb]$ cat /etc/passwd |grep jacknobody
[root@proddb proddb]$ 

What to do now?

Continue reading

EMagent 12c with high number of threads causing “su: cannot set user id: Resource temporarily unavailable”

Hello all,
Yeah, these days I got some errors and when validating the server found the fllowing error:

su: cannot set user id: Resource temporarily unavailable

As you can imagine, in order to fix the issue, I adjusted the /etc/security/limits.conf increasing oracle nprocs to:

oracle           soft    nproc           4047
oracle           hard    nproc           20384

Ok, turns out that after a while I got the same errors again…
After some investigating I find that the EM Agent process was with 5020 threads!
Take a look:

Continue reading

How To Delete Archive Logs From Downstream Database

After deploy GoldenGate with downstream database option archives from downstream database not cleaning ? All database transaction are gracefull, but archives (from downstream) are not cleaning?

Are no log transactions registered on GG Extract?
check this:

GGSCI (dbcloud) 3> send ETL01 showtrans

Sending SHOWTRANS request to EXTRACT ETL01 ...

Oldest redo log files necessary to restart Extract are:
XID:                  14.6.3931092
Items:                1
Extract:              ETL01
Redo Thread:          1
Start Time:           2017-00-00:00:00:00
SCN:                  1682.4049305132 (7228184297004)
Redo Seq:             8612
Redo RBA:             20965491728
Status:               Running

If you on same case, make this:

It should reach (and delete) JUST archived log that have already read by GoldenGate.

set serveroutput on size unlimited
set line 1000
set trimsp on
set feed off
set pages 5000
set pagesize 0
spool [[ some_dir ]]/

-- SR 3-14409179111 - Golden Gate Configuration, How To Delete Archive Logs From Downstream Database (Doc ID 2011174.1)
SELECT 'rm ' || r.NAME
-- case when (r.next_scn > c.required_checkpoint_scn) then 'NO' else 'YES' end purgable
WHERE r.CONSUMER_NAME = c.CAPTURE_NAME and r.source_database = c.source_database and r.next_scn < ( select min(required_checkpoint_scn) from dba_capture where captur
and purgeable='YES'
order by modified_time;
spool off

Schedule the output in automation tool (crontab / dba_job / windows job scheduler, (whatever)):

GOLDENGATE process show RUNNING status, when not RUNNING.

This might occur in case of crash on host or database node where goldengate is running.

Goldengate maintains information about the process status in the file system, when the server crashes the file that has PID data is not released, and the process status in GGSCI shows erroneous information.

In GGSCI, process status it’s OK:

GGSCI ( 7> info mgr

Manager is running (IP port, Process ID 42064).

If you try to stop process, you will not be able to:

Continue reading

Shellscript: Which database is this service_name from?

Strange question, right? You should know you applications and databases and how everything works, right?

Happen that some days ago a client requested me to identify from which database is a service_name. Easy? Not if the server have 46 database instances running…

Ok, now how can I can this information as quick as possible?

I know that the command below give me all distinct databases in CRS:

/u01/app/ stat res -t -w "TYPE = ora.database.type"

Now, lets make some sed magic:

[root@nprd01dbadm01 ~]# /u01/app/ stat res -t -w "TYPE = ora.database.type" |grep ora. |sed 's/ora./srvctl status service -d /' |sed 's/.db/ |grep \$1/' | head -1
srvctl status service -d dba |grep $1

Continue reading

GoldenGate Integrated Capture and Integrated Replicat Healthcheck Script

GoldenGate integrated Extract gives to dbas powerful tool to check GoldenGate’s operation in database, this package can  be found to download on Doc ID 1448324.1.

This Healthcheck is similar AWR reports and it been very useful to find some error or bottleneck.

Tool give some advices and parameter tips.

Let’s check my lab HC topics

Environment overview:lab1

Performance tips:



This HC uses system views created by OGG, so you can customize you own HC 🙂


Backup Not Backuped Archivelogs and Delete Input

Hi all!
Sometimes you are caught in a situation where your database is not backuping archivelogs and need to generate a quick backup commands for those are not backuped yet and deleting it, right?
I saw this situation in this archived discussion at OTN. Unfortunately I couldn’t give my answer… But it’s how I do:

'backup archivelog from sequence '||sequence#||' until sequence '||sequence#||' thread '||thread#||' filesperset=1 delete input;',first_time
from v$archived_log where backup_count=0 and name is not null
order by first_time desc;

It generates an output like:

greporadb> select
  2  'backup archivelog from sequence '||sequence#||' until sequence '||sequence#||' thread '||thread#||' filesperset=1 delete input;',first_time
  3  from v$archived_log where backup_count=0 and name is not null
  4  order by first_time desc;
backup archivelog from sequence 152153 until sequence 152153 thread 1 filesperset=1 delete input;                                                                       
backup archivelog from sequence 152152 until sequence 152152 thread 1 filesperset=1 delete input;                                                                       
backup archivelog from sequence 152151 until sequence 152151 thread 1 filesperset=1 delete input;                                                                       
3 rows selected.

And be happy!

But an observation! It not works this way for databases with dataguard. For these cases you’ll need to add “and name’&dgname’” at select where clause… 😉

See ya!