Weblogic in debug mode

Usually, in non-production environments, it is necessary to check applications deployed on a Weblogic server. The default log (.out) does not report or details conclusively the real cause of the problem.
In this case, beyond the levels of logs that can be configured via weblogic console (Managed Server > Logging > Advanced), we can add to the JVM startup arguments (Managed Server > Configuration > Server Start > Arguments) the following arguments:

-Dweblogic.webservice.verbose=true -Dweblogic.wsee.verbose=* -Dweblogic.wsee.verbose=weblogic.wsee.* -Dweblogic.wsee.verbose.timestamp=true

Recommended use only during the troubleshoot, because it generates a lot of logs.


Quickly change Weblogic to Production Mode

You were running away to deploy your newest project on Weblogic 12c and lately discover  that you made your environment as development mode (OPSSSS =/)

Quickly set check box ‘Production mode’ on your domain tab.

It will be necessary to bounce Weblogic server.


Thank’s Oracle for this checkbox in 12c 😛


Access denied on GoldenGate Manager

After apply GoldenGate fix on GoldenGate for Oracle Databases 11G getting error below during GoldenGate Director Server access:

ggserr.log: WARNING OGG-00936 Oracle GoldenGate Manager for Oracle, mgr.prm: Access denied (request from, rule #4).
To allow a remote Director server connection, you must add in the ./GLOBALS:


<EJB Exception in method: ejbPostCreate: java.sql.SQLException: XA error: XAResource.XAER_RMFAIL start() failed on resource 'ggds-datasource_domain': XAER_RMFAIL : Resource manager is unavailable

Some incidents that we face are expected. Usually we wait for problems when something changes in an environment.
But, some times, for no apparent reason, with no systemic alteration, we encounter errors where our first reaction is: what a f ***!?

This time we find a java exception in a standard domain for GoldenGate Director;

For months the application behaved stable and functional, until it did fails for no apparent reason;

When I saw part of the exception, “XAER_RMFAIL: Resource manager is unavailable” went straight to talk to one of the best DBAs know that Matheus Boesing, to request a check on the resource manager database, (No problem found).

… then we fall back into a BUG: 11672297 Bug: ORA-01092 MAPPED TO XAER_RMERR instead of XAER_RMFAIL – (Doc ID 1329800.1)

In version 12.1, this bug is fixed, but as a palliative solution can do the following:

Increase the value (Maximum Duration of XA Calls) in JTA configurations of weblogic domain, the default value is 12000, in my case, I adjusted to 48000;

The problem was solved, at least for now.

OGG-0352: Invalid character for character set UTF-8 was found while performing character validation of source column

Almost a month without post!
My bad, december is always a crazy time to DBAs, right?

This post’s title error happens because the charset is different between databases used on replication thought GoldenGate and occurs only with alphanumerical columns (CHAR, VARCHAR, VARCHAR2), because, even the char length be the same, the data length will be different (like I explained here). Take a look in this example:

sourcedb> select table_name,column_name,data_length,char_length from dba_tab_cols where column_name=’NAME' order by 1,2,3;
------------- --------------- ----------- -----------
TABLE_EXAMPLE NAME            25          25
destdb> select table_name,column_name,data_length,char_length from dba_tab_cols where column_name=’NAME' order by 1,2,3;

------------- --------------- ----------- -----------
TABLE_EXAMPLE NAME            100         25


There is basically two solutions:
1) Change one of those charsets.
2) Add “SOURCECHARSET PASSTHRU” clause on the replicat file.

I usually prefer the second option, just because it’s less intrusive than number 1.


See ya!

WLST easeSyntax

Who works with WLST know it’s pretty boring to natigate to MBeans, because whenever necessary to put in parentheses () commands and quotation marks ‘ ‘. When we forget, need to retype the whole command again.
I found a command that helps a lot when it comes to navigate in MBean tree, it eliminates the need for parentheses and quotation marks.
After entering the WLST, type:

wls:/xpto_domain/serverConfig> easeSyntax()

wls:/xpto_domain/serverConfig> ls
dr– AdminConsole

dr– SelfTuning
dr– Servers
dr– ShutdownClasses
dr– SingletonServices

wls:/xpto_domain/serverConfig> cd Servers
wls:/xpto_domain/serverConfig/Servers> ls
dr– AdminServer
dr– WLS1_MSWS1
dr– WLS1_MSWS2

wls:/xpto_domain/serverConfig/Servers> cd WLS1_MSWS1
wls:/xpto_domain/serverConfig/Servers/WLS1_MSWS1> cd Log
wls:/xpto_domain/serverConfig/Servers/WLS1_MSWS1/Log> cd ..
wls:/xpto_domain/serverConfig/Servers/WLS1_MSWS1> cd Machine
wls:/xpto_domain/serverConfig/Servers/WLS1_MSWS1/Machine> ls
dr– app1wsmachine1

Not tested within python scripts, only browsing the tree Mbean.


WebLogic AdminServer Startup stopped at “Initializing self-tuning thread pool”

After starting AdminServer, it remains with starting status and stopped writing in log file in:

Check the disk space used, to make sure that there are no partitions with 100% utilization, including /tmp.
After them, make sure the owner of the weblogic (oracle) has have write permission of “/tmp”

[root@app1xptoosb1 /]# ls -tlhr / |grep tmp
drwxr-xr-x 5 root root 4.0K Nov 15 09:11 tmp

If the owner of weblogic does not have write permission must be set, because the application server writes some temporary files in the directory:

[root@app1xptoosb1 /]# chmod 777 /tmp

[root@app1xptoosb1 /]# ls -tlhr / |grep tmp
drwxrwxrwx 10 root root 4.0K Nov 18 09:44 tmp


OGG-01934 Datastore repair failed, OGG-01931Datastore ‘dirbdb’ cannot be opened

After change GoldenGate 12c to ACFS filesystem, got eternal WARNING OGG-01931, even Datastore is created:
WARNING OGG-01931 Oracle GoldenGate Manager for Oracle, mgr.prm: Datastore ‘dirbdb’ cannot be opened. Error 2 (No such file or directory).

Complete solution steps to resolve the issue on SHARED FILESYSTEM:
1. Login to ggsci prompt and stop all OGG processes including jagent and manager
2. Run “delete datastore” command. Confirm delete of datastore
4. Start all OGG processes; start manager, start *, start jagent

Indicates that the data store should use System V shared memory for interprocess communications.


GoldenGate: RAC One Node Archivelog Missing

The situation:

We have a GoldenGate on Allow Mode running some extracts on RAC One Node Database (reading the archivelogs). And then, suddenly, the instance crashes (network lost contact to the server) and the other instance (thread) was auto started by CRS. To the database no problems: The other node redologs was used during the startup recover and every thing is ok.

The application running with Weblogic serverpool and gridlink just had a little contention and continued the operation thought the started instance. The Goldengate switch was manually made, but some sequences was lost. What we found? the sequences was in the old thread’s redologfiles. It should be backed up if fast_start_mttr_target was different to zero. Buuut, the world is not so beautiful:

raconenodedb> show parameters mttr
fast_start_mttr_target integer 0

How we solved?
Simple solution: identified the group/thread and made a cp from ASM. The copied redolog was used as archivelog on goldengate and everything was ok.

raconenodedb> select sequence#,group#,thread# from v$log where thread#=2 order by 1;
---------- ---------- ----------
39636 6 2
39637 7 2
39638 8 2
39639 9 2
39640 10 2
ASMCMD> cp group_10.288.859482805 /oracle/grup10_thread2
copying +DGDATA/MYDB/ONLINELOG/group_10.288.859482805 -> /oracle/grup10_thread2

Easy like that.