AIX: “WARNING: Heavy swapping observed on system in last 5 mins.”

Quick one today!

Having message below in your 11.2.0.3 on AIX like this?

WARNING: Heavy swapping observed on system in last 5 mins. 
pct of memory swapped in [31.28%] pct of memory swapped out [3.81%]. 
Please make sure there is no memory pressure and the SGA and PGA are configured correctly. 
Look at DBRM trace file for more details.

Stand down, this issue is caused by unpublished Bug 11801934, mentioned in MOS False Swap Warning Messages Printed To Alert.log On AIX (Doc ID 1508575.1).

Basically happens because the v$osstat does not reflect proper stats for the swap space paging.

So, stay calm and see you next week!

Searching entries on Alert.log: A Better Way

Hi all!
As the oldest readers know, someday I had to found some entries in the alertlog and I had a really big log. So I jerrry-ringed some scripts for grepping alert with auxiliar files and etc.
I can see the posts here: Grepping Alert by Day  and Grepping Entries from Alert.log.

So… They are functional, but probably the worst ways to get it. I didn’t know and was innocent to not search by the view x$dbgalertext.
There is also possible to write on alert through the procedure SYS.DBMS_SYSTEM.KSDWRT.

Ok, so let me fix this situation with theese two good guys: @write_alert and @find_alert

More“Searching entries on Alert.log: A Better Way”

ORA-06512 ORA-48168 ORA-12012 for ADR Job Raising Errors

ORA-06512 ORA-48168 ORA-12012 for ADR Job Raising Errors
A database is raising stack below on alertlog:

Errors in file /db/u7011/oracle/admin/MYDB/trace/MYDB_j002_22935.trc:
ORA-12012: error on auto execute of job "SYS"."DRA_REEVALUATE_OPEN_FAILURES"
ORA-48168: the ADR sub-system is not initialized
ORA-06512: at "SYS.DBMS_IR", line 522

More“ORA-06512 ORA-48168 ORA-12012 for ADR Job Raising Errors”

ADRCI Retention Policy and Ad-Hoc Purge Script for all Bases

As you know, since 11g we have a Automatic Diagnostic Repository (ADR). To better manage it, we also have a Command-line Interface, called ADRCI.
ADR contains all diagnostic information for database (logs, traces, incidents, problems, etc).

adr1

ADR Structure

More“ADRCI Retention Policy and Ad-Hoc Purge Script for all Bases”

Alert Log: “Private Strand Flush Not Complete” on Logfile Switch

Hi all!
Just a curiosity: Have you ever noticed in a database alert log the occourance of the following message for every logfile switch:

Thread 1 cannot allocate new log, sequence 9281
Private strand flush not complete
  Current log# 5 seq# 9280 mem# 0: /db/u5001/oradata/GREPORADB/redo05a.log
Thread 1 advanced to log sequence 9281 (LGWR switch)
  Current log# 6 seq# 9281 mem# 0: /db/u5001/oradata/GREPORADB/redo06a.log

More“Alert Log: “Private Strand Flush Not Complete” on Logfile Switch”

Getting today’s Errors and Warnings from MySQL log

Quick one!

# Warnings

cat /var/log/mysqld.log |grep `date +%y%m%d` | grep "\[Warning\]"

# Errors

cat /var/log/mysqld.log |grep `date +%y%m%d` | grep "\[ERROR\]"

And a Bonus!
To get entries from X days ago:

cat /var/log/mysqld.log |grep `date --date="46 days ago" +%y%m%d`

Matheus.

ORA-02062: distributed recovery

# Error/Alert

Errors in file /oracle/diag/rdbms/mydb/mydb2/trace/mydb2_reco_26083546.trc:
ORA-02062: distributed recovery received DBID e450df78, expected 0311e884

# Solution

begin
commit;
for d in (select local_tran_id from dba_2pc_pending) loop
dbms_transaction.purge_lost_db_entry( d.local_tran_id );
commit;
end loop;
end;
/

Matheus.

ORA-01548: active rollback segment found, terminate

# Problem

SQL> drop tablespace UNDOTBS1;
drop tablespace UNDOTBS1
*
ERROR at line 1:
ORA-01548: active rollback segment '_SYSSMU10_1251904955$' found, terminate dropping tablespace
SQL> drop rollback segment "_SYSSMU3_1251904955$";
Rollback segment dropped.
SQL> drop tablespace UNDOTBS1;
drop tablespace UNDOTBS1
*
ERROR at line 1:
ORA-01548: active rollback segment '_SYSSMU10_1251904955$' found, terminate dropping tablespace


# Solution

CREATE ROLLBACK SEGMENT rb1 STORAGE(INITIAL 1M next 1M minextents 20) tablespace UNDOTBS5;
CREATE ROLLBACK SEGMENT rb2 STORAGE(INITIAL 1M next 1M minextents 20) tablespace UNDOTBS5;
CREATE ROLLBACK SEGMENT rb3 STORAGE(INITIAL 1M next 1M minextents 20) tablespace UNDOTBS5;

# Why?
The UNDO_MANAGEMENT is set as ‘MANUAL’, right? To drop any undo the default UNDO must have at least one segment.

Matheus.

ORA-10456: cannot open standby database; media recovery session may be in progress

Easy, easy… Take a look:

# Error

db2database2p:>srvctl status database -d database
Instance database1 is running on node db1database1p
Instance database2 is not running on node db2database2p
db2database2p:>srvctl start instance -d database -i database2
PRCR-1013 : Failed to start resource ora.database.db
PRCR-1064 : Failed to start resource ora.database.db on node db2database2p
CRS-5017: The resource action "ora.database.db start" encountered the following error:
ORA-10456: cannot open standby database; media recovery session may be in progress
. For details refer to "(:CLSN00107:)" in "/grid/product/11.2.0/log/db2database2p/agent/crsd/oraagent_oracle/oraagent_oracle.log".
CRS-2674: Start of 'ora.database.db' on 'db2database2p' failed.

# Solution

db2database2p:>sqlplus / as sysdba
SQL*Plus: Release 11.2.0.3.0 Production on Thu Jun 4 20:27:46 2015
Copyright (c) 1982, 2011, Oracle. All rights reserved.

Connected to an idle instance.

SQL> startup
ORACLE instance started.
Total System Global Area 1.1224E+11 bytes
Fixed Size 2234920 bytes
Variable Size 6.1472E+10 bytes
Database Buffers 5.0466E+10 bytes
Redo Buffers 299741184 bytes
Database mounted.
ORA-10456: cannot open standby database; media recovery session may be in progress

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;
Database altered.
SQL> ALTER DATABASE OPEN READ ONLY;
Database altered.
SQL> exit

Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options

db2database2p:>srvctl status database -d database
Instance database1 is running on node db1database1p
Instance database2 is running on node db2database2p

Matheus.

Grepping Entries from Alert.log

Hey hey,
One more McGayver by me! Haha
Again to find some information in alert. This time, I’m looking to count and list all occurrences of an action in alert. To archive this, I made the script below.

grep-swiss-knife-590x295

The functionality is just a little bit more complex than the script of the last post, but stills quite simple. Take a look:

Parameters:
PAR1: name of alert (the main alert.log)
PAR2: Searched token
PAR3: Start day you want to, in the format “Mon dd” or just “Mon”. Below an example.
PAR4: Start Year (4 digits)
PAR5: [optional]End day you want to, in the format “Mon dd” or just “Mon”. The default value is “until now”.
PAR6: [optional]End Year (4 digits). The default value is “until now”. If you use the PAR5, you have to use PAR6.
PAR7: [optional] List All entries and when?. If you want to use this PAR, you must to use PAR5 and PAR6.

Examples (Looking for service reconfigurations):
Ex1: sh grep_entries_alert.sh alert_xxdb_1.log “services=” “Apr 12” 2015
(Seach between April 12 and now and count entries).
Ex2: sh grep_entries_alert.sh alert_xxdb_1.log “services=” “Apr 01” 2015 “May 30” 2015
(Seach between April 01 and May 30 and count the entries).
Ex3: sh grep_entries_alert.sh alert_xxdb_1.log “services=” “Apr 01” 2015 “May 30” 2015 LIST
(Seach between April 01 and May 30 and count the entries and list them all…)

# Script grep_entries_alert.sh
if [ $# -lt 6 ]; then
FIN=`cat $1 |wc -l`
else FIN=`cat $1 |grep -n $5 |grep $6$ |head -n 1 |cut -d':' -f1`
fi
BEG=`cat $1 |grep -n "$3" |grep $4$ |head -n 1 |cut -d':' -f1`
NMB=`expr $FIN - $BEG`
ENTR=`cat $1 |head -n $FIN |tail -$NMB| grep $2|wc -l`
echo Number of Entries: $ENTR >log.log
if [ $# -lt 7 ]; then
echo ------- Complete List Of Entries and When ---------- >> log.log
for line in `cat $1 |head -n $FIN |tail -$NMB| grep -n $2|cut -d':' -f1`;do
LR=`expr $line + $BEG` # To get "real line", without the displacement
DAT=`expr $LR - 1`     # To get line date of entry
echo awk \'NR==$DAT\' $1 >>aux.sh # Printing the lines just calculted
echo awk \'NR==$LR\' $1 >>aux.sh  # with aux.sh
done;
sh aux.sh >>log.log
fi
cat log.log

It’s not beautiful. But it works! 🙂

After that, there is the new blog sponsor:
MacGyver-macgyver-880400_200_228
(Hahahaha)

Matheus.