RHEL7: rc.local service not starting

It’s very common to automate application startup in rc.local on Linux systems.

Was testing Red Hat 7.2 (Maipo), and found that apps was’t started.

Found this on some Red Hat blog: 

Systemd is a system and service manager for Linux operating systems. It is designed to be backwards compatible with SysV init scripts, and provides a number of features such as parallel startup of system services at boot time, on-demand activation of daemons, support for system state snapshots, or dependency-based service control logic. In Red Hat Enterprise Linux 7, systemd replaces Upstart as the default init system.”

On default /etc/rc.local comes useful info:

#!/bin/bash
# THIS FILE IS ADDED FOR COMPATIBILITY PURPOSES
#
# It is highly advisable to create own systemd services or udev rules
# to run scripts during boot instead of using this file.
#
# In contrast to previous versions due to parallel execution during boot
# this script will NOT be run after all other services.
#
# Please note that you must run ‘chmod +x /etc/rc.d/rc.local’ to ensure
# that this script will be executed during boot.

touch /var/lock/subsys/local

Then, this ‘chmod’ turns rc.local enable during system startup.

 

To familiarize this new feature:

[root@somesystem~]# systemctl status rc-local
● rc-local.service - /etc/rc.d/rc.local Compatibility
Loaded: loaded (/usr/lib/systemd/system/rc-local.service; static; vendor preset: disabled)
Active: active (running) since Mon 2016-07-11 13:16:18 BRT; 28min ago
Process: 1046 ExecStart=/etc/rc.d/rc.local start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/rc-local.service
├─2272 /bin/sh /oracle/domains/mywl_domain/startWebLogic.sh
├─2284 /bin/sh /oracle/domains/mywl_domain/bin/startWebLogic.sh
├─2374 /bin/sh /oracle/domains/mywl_domain/bin/startNodeManager.sh
├─2377 /bin/sh /oracle/binaries/wlserver/server/bin/startNodeManager.sh
├─2428 /oracle/jdk1.7.0_25/bin/java -Dwls.home=/oracle/binaries/wlserver/server -Dweblogic.home=/oracle/binaries/wlserver/server -server -Xms1g -Xmx1g -XX:MaxPermSize=512m -Dcoherence...
└─2442 /oracle/jdk1.7.0_25/bin/java -server -Xms1g -Xmx1g -XX:MaxPermSize=512m -Dweblogic.Name=AdminServer -Djava.security.policy=/oracle/binaries/wlserver/server/lib/weblogic.policy .

Maiquel.
 

Weblogic starting with the operating system

Hi,
Today, let’s to configure weblogic services startup, when machines starts.
In some blogs, we can find a bunch of customized scripts that create and set variables to startup the adminservers, nodemanagers and managed server, but, in my case, i just need to start adminserver and nodemanger, when machines start just after an incident.

For this situation, we need that the startup of application do not interrupt the operation system startup.

*The operation system in subject is Red-Hat 6.5

More“Weblogic starting with the operating system”

Changing ACFS mount point

I do checked there’s no good way to change ACFS mounting point on asmca assistant, so I decided to document how I quickly change ACFS mount point:

    1. MAKE BACKUP ( in my case, there are no data loss );
    2. Do bellow:
root@mymachine:/oracle/product >/grid/product/12.1.0.2/bin/srvctl stop filesystem -d /dev/asm/ggatebin-68

root@mymachine:/ >/usr/sbin/acfsutil registry -d /dev/asm/ggatebin-68
acfsutil registry: successfully removed ACFS volume /dev/asm/ggatebin-68 from Oracle Registry

root@mymachine:/ >/usr/sbin/acfsutil registry -a /dev/asm/ggatebin-68 /oracle/product/goldengate12c/
acfsutil registry: mount point /oracle/product/goldengate12c successfully added to Oracle Registry

root@mymachine:/oracle/product >chown -R oracle.oinstall goldengate12c
root@mymachine:/oracle/product >chmod 755 goldengate12c

Maiquel.

RMAN Raise ORA-19913 ORA-28365 On Restore from Cloud Backup

First I think was some error with Database Backup To Cloud, when testing. Then I realized it was a simple mistake by myself.

Let me show you. First trying to restore datafile:

[oracle@mydbsrvr archivelogs]$ rman target /
RMAN> restore datafile 6;
Starting restore at 03-MAY-2016 20:00:30
using channel ORA_SBT_TAPE_1
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=178 device type=DISK
channel ORA_SBT_TAPE_1: starting datafile backup set restore
channel ORA_SBT_TAPE_1: specifying datafile(s) to restore from backup set
channel ORA_SBT_TAPE_1: restoring datafile 00006 to /db/u1001/test/cloud_test/test_restore.dbf
channel ORA_SBT_TAPE_1: reading from backup piece 0sr4mdun_1_1
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of restore command at 05/03/2016 20:00:34
ORA-19870: error while restoring backup piece 0sr4mdun_1_1
ORA-19913: unable to decrypt backup
ORA-28365: wallet is not open

Ok, it might happen because I forgot to set encryption password:

RMAN> SET ENCRYPTION ON IDENTIFIED BY "matheusdba" only;
executing command: SET encryption
RMAN> restore datafile 6;
Starting restore at 03-MAY-2016 20:00:30
using channel ORA_SBT_TAPE_1
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=178 device type=DISK
channel ORA_SBT_TAPE_1: starting datafile backup set restore
channel ORA_SBT_TAPE_1: specifying datafile(s) to restore from backup set
channel ORA_SBT_TAPE_1: restoring datafile 00006 to /db/u1001/test/cloud_test/test_restore.dbf
channel ORA_SBT_TAPE_1: reading from backup piece 0sr4mdun_1_1
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of restore command at 05/03/2016 20:00:34
ORA-19870: error while restoring backup piece 0sr4mdun_1_1
ORA-19913: unable to decrypt backup
ORA-28365: wallet is not open

It hapen again?
This point I suspect some kind of bug… But it was my mistake and is not related to Cloud, but to Encryption use. To undestand:
For Backup: Use ENCRYPTION
For Restore/Recover: Use DECRYPTION

Obviously, but take me a minute to realize…

Setting decryption, and problem solved:

RMAN> set DECRYPTION identified by "matheusdba";
executing command: SET decryption
RMAN> restore datafile 6;
Starting restore at 03-MAY-2016 20:00:58
using channel ORA_SBT_TAPE_1
using channel ORA_DISK_1
channel ORA_SBT_TAPE_1: starting datafile backup set restore
channel ORA_SBT_TAPE_1: specifying datafile(s) to restore from backup set
channel ORA_SBT_TAPE_1: restoring datafile 00006 to /db/u1001/test/cloud_test/test_restore.dbf
channel ORA_SBT_TAPE_1: reading from backup piece 0sr4mdun_1_1
channel ORA_SBT_TAPE_1: piece handle=0sr4mdun_1_1 tag=TAG20160503T193239
channel ORA_SBT_TAPE_1: restored backup piece 1
channel ORA_SBT_TAPE_1: restore complete, elapsed time: 00:00:03
Finished restore at 03-MAY-2016 20:01:02

See ya!
Matheus.

UnknownHostException: Could not authenticate to Oracle Database Cloud Backup Module

Hi all!
When running Oracle Database Cloud Backup Module, found this error:

Command:

java -jar opc_install.jar -serviceName Storage -identityDomain usmatheusdba -opcId 'matheus@boesing.com.br' -opcPass 'BestBlog2016' -walletDir /db/oracle/admin/cloud/wallet -libDir /db/oracle/admin/cloud/libs

(Credential values changed, of course…)

Error:

Oracle Database Cloud Backup Module Install Tool, build 2016-02-04
java.net.UnknownHostException: usmatheusdba.storage.oraclecloud.com
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:175)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:385)
at java.net.Socket.connect(Socket.java:546)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:602)
at sun.security.ssl.BaseSSLSocketImpl.connect(BaseSSLSocketImpl.java:160)
at sun.net.NetworkClient.doConnect(NetworkClient.java:178)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:427)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:529)
at sun.net.www.protocol.https.HttpsClient.(HttpsClient.java:275)
at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:332)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:891)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1226)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:254)
at oracle.backup.opc.install.OpcConfig.testConnection(OpcConfig.java:235)
at oracle.backup.opc.install.OpcConfig.doOpcConfig(OpcConfig.java:204)
at oracle.backup.opc.install.OpcConfig.main(OpcConfig.java:197)
Could not authenticate to Oracle Database Cloud Backup Module

Solution:
Set Relication Policy of Oracle Storage Cloud Service.
In My Services Home, Oracle Storage Cloud Service will have a link to “Set Retention Policy”. It’s simply set it.
But pay attention, once you select a replication policy, you can’t change it.

As you can see, I already did it:
retention_policy

After that, everything worked fine. 🙂

KB:
Problems with Installing the Backup Module
Selecting a Replication Policy for Oracle Storage Cloud Service

See ya!
Matheus.

kernel.panic_on_oops: New Oracle 12c Installation Requirement

Hi all,
Do you know what mean the parameters on installing 12c?

This parameter controls the kernel’s behaviour when an oops or bug is encountered:

  • 0: try to continue operation
  • 1: panic immediately.  If the `panic’ sysctl is also non-zero then the machine will be rebooted.

OOPS is a deviation from correct behavior of the Linux kernel, one that produces a certain error log.
The better-known kernel panic condition results from many kinds of oops, but other instances of an oops event may allow continued operation with compromised reliability.

This is recommended in a system where we want to have node evicted in case of any hardware failure or any other issue.

To adjust as recommended by Oracle?
1. Put an entry in sysctl.conf for having it permanent:

kernel.panic_on_oops = 1

2. Refresh running command:

sysctl -p

KB: https://www.kernel.org/doc/Documentation/sysctl/kernel.txt

Matheus.

Oracle Database: Compression Algorithms for Cloud Backup

Hi all!
Again talking about cloud backups for on-premise databases: An important aspect is to compress the data, so network consumption might be reduced once less data is being transfered.Compress_PDF
It’s also important to evaluate CPU consumption. As higher compress algorithm is, as much CPU it uses. So, pay attention!

Now, how to choose the compression algorithm? Here the options Oracle give us:

SQL> col ALGORITHM_NAME for a15
set line 200
SQL> select ALGORITHM_NAME,INITIAL_RELEASE,TERMINAL_RELEASE,ALGORITHM_DESCRIPTION,ALGORITHM_COMPATIBILITY  from v$rman_compression_algorithm;
ALGORITHM_NAME INITIAL_RELEASE    TERMINAL_RELEASE   ALGORITHM_DESCRIPTION                                            ALGORITHM_COMPATIB
-------------- ------------------ ------------------ ---------------------------------------------------------------- ------------------
BZIP2          10.0.0.0.0         11.2.0.0.0         good compression ratio                                           9.2.0.0.0
BASIC          10.0.0.0.0                            good compression ratio                                           9.2.0.0.0
LOW            11.2.0.0.0                            maximum possible compression speed                               11.2.0.0.0
ZLIB           11.0.0.0.0         11.2.0.0.0         balance between speed and compression ratio                      11.0.0.0.0
MEDIUM         11.2.0.0.0                            balance between speed and compression ratio                      11.0.0.0.0
HIGH           11.2.0.0.0                            maximum possible compression ratio                               11.2.0.0.0

How to identify our compression algorithm?

RMAN> show COMPRESSION ALGORITHM;
RMAN configuration parameters for database with db_unique_name EZM_PRFL are:
CONFIGURE COMPRESSION ALGORITHM 'BASIC' AS OF RELEASE 'DEFAULT' OPTIMIZE FOR LOAD TRUE ; # default

And how to change it?

RMAN> CONFIGURE COMPRESSION ALGORITHM 'HIGH';
new RMAN configuration parameters:
CONFIGURE COMPRESSION ALGORITHM 'HIGH' AS OF RELEASE 'DEFAULT' OPTIMIZE FOR LOAD TRUE;
new RMAN configuration parameters are successfully stored
RMAN> show COMPRESSION ALGORITHM;
RMAN configuration parameters for database with db_unique_name EZM_PRFL are:
CONFIGURE COMPRESSION ALGORITHM 'HIGH' AS OF RELEASE 'DEFAULT' OPTIMIZE FOR LOAD TRUE;

Ok,
But how to evaluate my compression ratio?

See the difference between INPUT_BYTES_DISPLAY and OUTPUT_BYTES_DISPLAY columns from the query:

prddb> col STATUS for a10
prddb> col INPUT_BYTES_DISPLAY for a15
prddb> col OUTPUT_BYTES_DISPLAY for a15
prddb> col TIME_TAKEN_DISPLAY for a20
prddb> SELECT SESSION_KEY,
2         INPUT_TYPE,
3         STATUS,
4         TO_CHAR(START_TIME, 'mm/dd/yy hh24:mi') start_time,
5         TO_CHAR(END_TIME, 'mm/dd/yy hh24:mi') end_time,
6  --       ELAPSED_SECONDS / 3600 hrs,
7         COMPRESSION_RATIO,
8         INPUT_BYTES_DISPLAY,
9         OUTPUT_BYTES_DISPLAY,
10         TIME_TAKEN_DISPLAY
11    FROM V$RMAN_BACKUP_JOB_DETAILS
12    where input_type like 'DB%'
13   ORDER BY SESSION_KEY
14  /SESSION_KEY INPUT_TYPE    STATUS     START_TIME     END_TIME       COMPRESSION_RATIO INPUT_BYTES_DIS OUTPUT_BYTES_DI TIME_TAKEN_DISPLAY
----------- ------------- ---------- -------------- -------------- ----------------- --------------- --------------- --------------------
          2 DB FULL       COMPLETED  04/22/16 12:59 04/22/16 13:06        6,84838963     4.26G         636.50M       00:06:57
          9 DB FULL       COMPLETED  04/22/16 13:47 04/22/16 13:54        6,83764706     4.26G         637.50M       00:06:37
         14 DB FULL       COMPLETED  04/22/16 16:26 04/22/16 16:33        6,84189878     4.26G         637.25M       00:06:48

KB: https://docs.oracle.com/cd/E11882_01/backup.112/e10642/rcmconfa.htm#BRADV89466

Done?
If you have any question, please let me know in the comments! 😉
Matheus.

Installing and Configuring ASMLIb on Oracle Linux 7

Hi all!
For those are familiar with RHEL/OEL 4 and 5, there is some differences to start ASMLib on OEL 6 and 7.

spanner.png
So, a quick guide to install (done on OEL 7), start and configure:

1. Install the ASMLib kernel module package as root using the following command:

yum install kmod-oracleasm

2. Install the ASMLib library package and utilities package

yum install oracleasm-support oracleasmlib oracleasm-`uname -r`

It’s possible some package to not found. For example:

No package oracleasmlib available.

So, you can download rpm libs from here and install via rpm:

[root@dbsrv01 oracle]# rpm -Uvh ~/oracleasmlib-2.0.12-1.el6.x86_64.rpm
Preparing...                          ################################# [100%]
Updating / installing...
1:oracleasmlib-2.0.12-1.el6        ################################# [100%]

Ok, now, lets configure/start services:

[root@dbsrv01 ~]# /etc/init.d/oracleasm configure

Nothing happen? Ok, let’s try to start it:

[root@dbsrv01 ~]# /etc/init.d/oracleasm start
Starting oracleasm (via systemctl):  Job for oracleasm.service failed because the control process exited with error code. See "systemctl status oracleasm.service" and "journalctl -xe" for details.
[FAILED]

Hmmm… Are these commands correct?

[root@dbsrv01 ~]# /etc/init.d/oracleasm
Usage: /etc/init.d/oracleasm {configure|createdisk|deletedisk|querydisk|listdisks|scandisks|status}

Ok… So, what to do?

Take a look:

[root@dbsrv01 ~]# oracleasm init
Creating /dev/oracleasm mount point: /dev/oracleasm
Loading module "oracleasm": oracleasm
Configuring "oracleasm" to use device physical block size
Mounting ASMlib driver filesystem: /dev/oracleasm

Victory!
Now, let’s configure:

[root@dbsrv01 ~]# oracleasm configure
ORACLEASM_UID=
ORACLEASM_GID=
ORACLEASM_SCANBOOT=true
ORACLEASM_SCANORDER=""
ORACLEASM_SCANEXCLUDE=""
ORACLEASM_USE_LOGICAL_BLOCK_SIZE="false"

It shows, but how configure?

Just put “-i” clause, like:

[root@dbsrv01 ~]# oracleasm configure -i
Configuring the Oracle ASM library driver.
This will configure the on-boot properties of the Oracle ASM library
driver.  The following questions will determine whether the driver is
loaded on boot and what permissions it will have.  The current values
will be shown in brackets ('[]').  Hitting  without typing an
answer will keep that current value.  Ctrl-C will abort.
Default user to own the driver interface []: grid
Default group to own the driver interface []: oinstall
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: done

And you can list again:

[root@dbsrv01 ~]# oracleasm configure
ORACLEASM_UID=grid
ORACLEASM_GID=oinstall
ORACLEASM_SCANBOOT=true
ORACLEASM_SCANORDER=""
ORACLEASM_SCANEXCLUDE=""
ORACLEASM_USE_LOGICAL_BLOCK_SIZE="false"
[root@dbsrv01 ~]# oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes

To add a disk, the same process can be followed on earlier versions:

[root@dbsrv01 ~]# oracleasm createdisk SDD /dev/sdd1
Writing disk header: done
Instantiating disk: done
[root@dbsrv01 ~]# oracleasm listdisks
SDD

For all commands:

[root@dbsrv01 ~]# oracleasm -h
Usage: oracleasm [--exec-path=]  [  ]
oracleasm --exec-path
oracleasm -h
oracleasm -V
The basic oracleasm commands are:
configure        Configure the Oracle Linux ASMLib driver
init             Load and initialize the ASMLib driver
exit             Stop the ASMLib driver
scandisks        Scan the system for Oracle ASMLib disks
status           Display the status of the Oracle ASMLib driver
listdisks        List known Oracle ASMLib disks
querydisk        Determine if a disk belongs to Oracle ASMlib
createdisk       Allocate a device for Oracle ASMLib use
deletedisk       Return a device to the operating system
renamedisk       Change the label of an Oracle ASMlib disk
update-driver    Download the latest ASMLib driver

And to see arguments for each one:

[root@dbsrv01 ~]# oracleasm configure -h
Usage: oracleasm-configure [-l ] [-i|-I] [-e|-d] [-u ] [-g ] [-b|-p] [-s y|n] [[-o ] ...] [[-x ] ...]

Have a nice day!
See ya!
Matheus.

nc -l – Starting up a fake service

Hi everyone!

Recently i have faced a situation that made me find out a very nice and useful command that helped me a lot, and i hope it comes to help you guys as well, and it’s named:

nc

Situation: We have a replicated environment from one datacenter to another (Using Golden Gate), where the ETL happens. So basically is:

Datacenter 1 (root data)

Replicates to datacenter 2 (transforming the data)

that replicates to datacenter 3 (production itself)

In Datacenter level 2, we have a dataguard configured. So then came the question:

  • What if we need to do the switchover to the standby environments?
  • Will we gonna have everything we need properly set up for the replication?
  • How are we going to test the ports if nothing is up in there? Aren’t we gonna get “connection refused”?

Calm down! There is a very nice workaround for this.

All you need to do is install the nc command as root (if it is not installed already):

yum install nc

Then execute it as follows, on the server you wanna test:

nc -l

example:

I wanna make sure that on the standby server the port 7809 (Golden Gate MANAGER port) is open. On the standby server you run:

nc -l 7809

Then, from a remote server, you are going to be able to connect through a simple telnet command:

telnet server.domain port

example:

telnet standby.company.com 7809

 

ON PRACTICE:

  • Try the telnet from the remote server to the standby:

remoteserver {/home/oracle}: telnet standby.server 7809

Trying 192.168.0.10…

telnet: connect to address 192.168.0.10: Connection refused

  • Then we start the fake service on the standby server!

standby.server {/home/oracle}: nc -l 7809

  • And try the telnet again:

remoteserver {/home/oracle}: telnet standby.server 7809

Trying 192.168.0.10…

Connected to standby.server.

Escape character is ‘^]’.

 

Cheers!

Rafael.

Oracle Database Backup to Cloud: KBHS – 01602: backup piece 13p0jski_1_1 is not encrypted

Hi all!
I’m preparing a material about downloading, configuring using Oracle Database Cloud Backup. My case is about backuping a local database to Cloud.

So, as avant-première for you from the Blog, a quick situation about:

cloud-backup.png

# Error

RMAN-03009: failure of backup command on ORA_SBT_TAPE_1 channel at 04/14/2016 13:58:45
ORA-27030: skgfwrt: sbtwrite2 returned error
ORA-19511: non RMAN, but media manager or vendor
specific failure, error text:
KBHS - 01602: backup piece 12p1krsi_1_1 is not encrypted

# Solution (one of)

RMAN> set encryption on identified by "mypassword" only;
executing command: SET encryption


Why?

To use Oracle Database Backup to Cloud you need to use at least one encrypting method.
Oracle offers basically 3:
– Password Encryption
– Transparent Data Encryption (TDE)
– Dual-Mode Encryption (a combination of password and TDE).

In this post I refered the easier, by I recommend you to take a look on KB: https://docs.oracle.com/cloud/latest/dbbackup_gs/CSDBB.pdf

Matheus.