Category: Cloud Services | Exadata | Linux

OS, Linux, Oracle Engineered Systems, Cloud Services.

Enterprise Manager, Linux

OEM Metric “Memory Utilization” Different on 12c and 13c

 , , ,

So, as rollout strategy we created a new OEM13c to decommission a 12c. However during the testes, noticed Memory Utilization metric was a lot different between 12c and 13c. Why?

Happens that the Memory Utilization is calculated differently between 12c and 13c, but also seems 13c is more accurate, as per MOS The Host Memory Utilization Percentage Calculation in Enterprise Manager Cloud Control (Doc ID 1908853.1)

Well, those who are familiar with memory use computations in the operating system might become confused when examining the memory use metric data from Enterprise Manager 12c and 13c Cloud Control. Metrics such as Memory Utilization (%) do not have an equivalent in the OS, but OS data will be used in its derivation.

This is the formula used by Enterprise Manager 12.1.0.3 for Linux Memory Utilization (%), for example:

Memory Utilization (%) = (100.0 * (activeMem) / realMem)
 = 100 * 25046000/99060536
 = 25.28
EM Shows : 25.5

* On this, activeMem is Active Memory (Active), and realMem is Total Memory (MemTotal).

Comparing this with MemFree, which is not valid, might provide an impression that utilization is not being accurately represented.

Also, the “OEM13c value” was already collected in OEM12c, but under metric name “Used Logical Memory”. And basically “Memory Utilization” in 12c uses “activeMem” instead of “realMem-(freeMem+Buffers+Cached)”. As per image below.

OEM12_grep_mem

The formula in place on 13c is exactly the same as used to fix MOS EM 13c: Incorrect Memory Utilization Reported for Linux Hosts in Enterprise Manager 13.1.0.0.0 Cloud Control (Doc ID 2144976.1)

Example:

[root@greporasrv ~]# free
             total       used       free     shared    buffers     cached
Mem:     264087460  257669460    6418000    7657500     461088   11008128
-/+ buffers/cache:  246200244   17887216
Swap:     25165820    3365104   21800716

(100.0 * (realMem-(freeMem+Buffers+Cached)) / realMem)
100*(264087460-(6418000+461088+11008128))/264087460) = 93,22678328

As per OEM13c:

OEM13_grep_mem.jpg

Also, by checking on server using SAR, seems value in OEM 13c is more accurate, indeed:

[root@greporasrv ~]# sar -r
Linux 2.6.39-400.294.4.el6uek.x86_64 (greporasrv) 	08/29/2017 	_x86_64_	(44 CPU)

12:00:01 AM kbmemfree kbmemused  %memused kbbuffers  kbcached  kbcommit   %commit
12:10:01 AM   5377540 258709920     97.96    719080  10775828  83876744     29.00
12:20:01 AM   6131220 257956240     97.68    719504  10721084  82467712     28.51
12:30:01 AM   5623060 258464400     97.87    719700  10720972  83456216     28.85
12:40:01 AM   5606572 258480888     97.88    719836  10779108  83228440     28.77
12:50:01 AM   5783256 258304204     97.81    719860  10848644  82925908     28.67
01:00:01 AM   4151148 259936312     98.43    719888  11589048  84400040     29.18
01:10:01 AM   3717000 260370460     98.59    719904  11534336  84838784     29.33
01:20:01 AM   4282412 259805048     98.38    720164  11480792  84047568     29.06
01:30:01 AM   4473128 259614332     98.31    720184  11483604  83857348     28.99
01:40:01 AM   5113136 258974324     98.06    720256  11528492  83036284     28.71
01:50:01 AM   4971036 259116424     98.12    720284  11587956  82955128     28.68
02:00:01 AM   4026540 260060920     98.48    720344  11663184  86489692     29.90
02:10:01 AM   4312916 259774544     98.37    720380  11678316  83834592     28.98
02:20:01 AM   5058980 259028480     98.08    720408  11624028  82876972     28.65
02:30:01 AM   4609908 259477552     98.25    720556  11541392  83871244     29.00
02:40:01 AM   5020668 259066792     98.10    720592  11574912  82887808     28.66
02:50:01 AM   5175916 258911544     98.04    720748  11619572  82725252     28.60
03:00:01 AM   4701236 259386224     98.22    720780  11687100  83421624     28.84
03:10:01 AM   4757976 259329484     98.20    721204  11648864  83298716     28.80
03:20:01 AM   4485280 259602180     98.30    721248  11719272  83299472     28.80
03:30:01 AM   4267068 259820392     98.38    721264  11794688  83683344     28.93
03:40:01 AM   4080264 260007196     98.45    721404  11856796  83863540     28.99
03:50:01 AM   4864276 259223184     98.16    721676  11975372  82735744     28.60
04:00:01 AM   4427284 259660176     98.32    721696  12056676  83450524     28.85
04:10:01 AM   4868184 259219276     98.16    721736  11863420  82860464     28.65
04:20:01 AM   4711608 259375852     98.22    721760  11877192  83205684     28.77
04:30:01 AM   4452764 259634696     98.31    721928  11945108  83515596     28.87
04:40:01 AM   4800700 259286760     98.18    722072  12015444  82681320     28.58
04:50:01 AM   4796588 259290872     98.18    722212  12075496  82703948     28.59
05:00:01 AM   4320164 259767296     98.36    722372  12164956  83390596     28.83
05:10:01 AM   3350940 260736520     98.73    722488  12120116  84525028     29.22
05:20:01 AM   4200236 259887224     98.41    722628  11965996  83510580     28.87
05:30:01 AM   4028020 260059440     98.47    722640  12019516  83720748     28.94
05:40:01 AM   3929740 260157720     98.51    722720  12069520  83632964     28.91
05:50:01 AM   2719452 261368008     98.97    723460  14408924  83745112     28.95
06:00:01 AM   1530448 262557012     99.42    723644  14943264  84618304     29.25
06:10:01 AM   2925268 261162192     98.89    605748  13363596  84792452     29.31
06:20:02 AM   3235532 260851928     98.77    605916  13811664  83516740     28.87
06:30:01 AM   3265640 260821820     98.76    606072  13848028  83385196     28.83
06:40:01 AM   2102756 261984704     99.20    606232  14745508  83638764     28.92
06:50:01 AM   2386376 261701084     99.10    606644  14821232  83118484     28.74
07:00:01 AM   5343496 258743964     97.98    186908  12019804  84375032     29.17
07:10:01 AM   5073472 259013988     98.08    219044  12597104  83579876     28.90
07:20:01 AM   5380380 258707080     97.96    241300  12600412  83107160     28.73
07:30:01 AM   5063504 259023956     98.08    253984  12653840  83373804     28.82
07:40:01 AM   8241032 255846428     96.88    269960   9772232  83072188     28.72
07:50:01 AM   8549616 255537844     96.76    278472   9853288  82646916     28.57
08:00:01 AM   8185864 255901596     96.90    287296   9938816  83179808     28.76
08:10:01 AM   7797504 256289956     97.05    295856  10029904  83464160     28.86
08:20:01 AM   8813696 255273764     96.66    302620   9930672  82081220     28.38
08:30:01 AM   8574984 255512476     96.75    309156   9880124  82557600     28.54
08:40:01 AM   8010072 256077388     96.97    314804   9912220  83241764     28.78
08:50:01 AM   8791112 255296348     96.67    319568   9980532  81787424     28.28
Database, Linux, Shell Scripts

Oracle memory usage on Linux / Unix

 , ,

Hi all,

So one of the most important things that we need to do when setting up a new server or checking the capacity of the server is to see how much memory Oracle is using.

When checking the capacity there are some practical things that always help me to get a fast glimpse of the system:

  • When opening topas and hitting M you will see this below
Topas Monitor for host: SERVER1 Interval: 2 Sat Dec 8 03:39:59 2019
================================================================================
REF1 SRAD TOTALMEM INUSE FREE FILECACHE HOMETHRDS CPUS
--------------------------------------------------------------------------------
0 0 59.8G 59.6G 212.3 16.3G 528 0-15
1 1 61.4G 61.2G 188.8 15.7G 536 16-31

On the memory session you will see 3 categories, INUSE, FREE and FILECACHE. There you may see what is being using for what but there is not much granularity there.

  • When using top you have this summary below
top - 11:48:08 up 119 days, 10:18, 1 user, load average: 26.76, 26.16, 25.95
Tasks: 1936 total, 38 running, 1898 sleeping, 0 stopped, 0 zombie
Cpu(s): 79.3%us, 1.1%sy, 0.0%ni, 15.1%id, 4.3%wa, 0.0%hi, 0.1%si, 0.0%st
Mem: 263750172k total, 219075656k used, 44674516k free, 797476k buffers
Swap: 16773116k total, 505760k used, 16267356k free, 88055108k cached

Same you have a high level usage. So here comes the question:

How are you to prove that you have a memory shortage?

I often use vmstat on Linux looking on the columns si and so equals to 0 (swap in and swap out) and when the free command, the free column you will also have no or very low swap being used

/home/oracle> vmstat 1
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
15 3 505760 44896608 797480 88062288 0 0 7037 1304 0 0 29 2 61 8 0
16 1 505760 44922964 797480 88062320 0 0 432272 144314 38784 31348 41 2 52 5 0
14 2 505760 44943072 797480 88062320 0 0 468904 155424 32676 27522 34 1 60 5 0
15 2 505760 44943032 797480 88062328 0 0 431032 144275 32596 27469 34 1 60 5 0
15 2 505760 44920136 797480 88062352 0 0 396232 145052 30772 26657 32 1 62 6 0
19 1 505760 44928576 797480 88062360 0 0 429360 160158 33640 28012 36 1 58 5 0
15 3 505760 44935340 797480 88062368 0 0 477232 161849 28393 21423 41 1 53 5 0
17 1 505760 44924744 797480 88062368 0 0 515265 160212 27478 20578 40 1 54 5 0
16 1 505760 44921596 797480 88062368 0 0 495408 159304 25458 19548 37 1 58 5 0
18 1 505760 44918144 797480 88062384 0 0 552880 168895 28203 22774 38 1 56 5 0
15 2 505760 44922344 797480 88062392 0 0 546920 160463 25321 19151 37 1 58 5 0
16 4 505760 44921544 797480 88062400 0 0 571544 153810 25429 20011 36 1 58 5 0
16 1 505760 44919620 797480 88062400 0 0 577552 160004 27132 20111 40 1 54 5 0
19 2 505760 44360240 797480 88062400 0 0 584969 155553 29467 22145 41 2 52 5 0
/home/oracle> free
total used free shared buffers cached
Mem: 263750172 219060896 44689276 91608 797480 88062464
-/+ buffers/cache: 130200952 133549220
Swap: 16773116 505760 16267356

To check a process specific memory usage (RSS) I often use ps along with other commands to calculate the process memory for a specific process id as below:

/home/oracle> ps -eo rss,pid,euser,lstart,args:100 --sort %mem | grep -v grep | grep 35796 | awk '{printf $1/1024 "MB"; $1=""; print }'| sort
19.6016MB 35796 oracle Sat Sep 8 02:43:54 2018 ora_lg00_ORC1
34.957MB 32340 oracle Sat Jan 5 11:50:09 2019 oracleORC1 (LOCAL=NO)

RSS is resident memory, but when comes to shared memory like the Oracle SGA the methods above could be miss leading – not to say wrong – but as Oracle memory is shared we may see double counting on the results. I sometimes use pmap to check a process memory as well when available

/home/oracle> pmap 35796
35796: ora_lg00_ORC1
total 0K

But, still when checking a server wide scope, do you want to keep doing manual work and lots of math? I don’t think so. 🙂

That’s why when I came across SMEM made my life a lot easier. It is a python script which gives you a nice breakdown of the memory usage and without the miss leading double counting.

You can see the commands and processes and their memory:

[root@srv01 smem-1.4]# ./smem -trk | head
PID User Command Swap USS PSS RSS
4829 root /opt/stackdriver/collectd/s 444.0K 4.0G 4.0G 4.0G
5647 oracle asm_gen0_+ASM 50.1M 424.4M 425.0M 437.8M
16512 oracle rman software/product/11.2. 0 172.9M 173.7M 177.8M
85107 oracle ora_n001_db01 42.3M 147.8M 147.8M 185.8M
85103 oracle ora_n000_db01 42.4M 146.5M 146.6M 184.6M
85109 oracle ora_n002_db01 42.2M 145.6M 145.6M 183.5M
85111 oracle ora_n003_db01 42.1M 145.1M 145.2M 183.1M
7287 oracle ora_dia0_db01 1.6M 68.6M 68.8M 107.8M

As well the overall server per user:

root@srv01 smem-1.4]# ./smem -turk 
User Count Swap USS PSS RSS oracle 1358 4.8G 7.8G 8.0G 76.6G 
root 43 12.0M 4.1G 4.1G 4.2G user1 10 0 321.0M 328.0M 369.2M 
nobody 2 96.0K 2.1M 2.3M 6.0M user2 2 0 684.0K 1.7M 7.7M 
user4 2 0 632.0K 1.7M 7.9M user4 1 72.0K 536.0K 540.0K 2.1M 
ntp 1 424.0K 332.0K 368.0K 2.4M 
smmsp 1 1.3M 160.0K 298.0K 1.9M 
rpc 1 336.0K 68.0K 73.0K 1.7M 
rpcuser 1 808.0K 4.0K 16.0K 1.9M 
--------------------------------------------------- 
1422 4.8G 12.2G 12.5G 81.3G

Hope it helps, see you next time!

Cloud Services | Exadata | Linux, Linux, Shell Scripts

Amazon EC2: X11 Forwarding After Sudo SSH Session

 , , , , ,

Hello all!

So, now with more use of resources like Cloud servers, more and more silent instalations are being done, right? Myself, I do it in silent always I can.

What if I need to export X. Plus, if I need to export it from user oracle, but I can only login with ec2-user, as usual?

Here is the process for that:

1) Connect to AWS EC2 instance

[user@securehost ~]$ ssh -X ec2-user@ipaddress
Last login: Fri Dec 7 14:41:41 2018 from grepora.srv.com
__| __|_ )
 _| ( / Amazon Linux AMI
 ___|\___|___|
https://aws.amazon.com/amazon-linux-ami/2018.03-release-notes/
13 package(s) needed for security, out of 16 available
Run "sudo yum update" to apply all updates.

2) Test xclock works from ec2-user

[ec2-user@ipaddress ~]$ xclock
Warning: Missing charsets in String to FontSet conversion
^C

3) Show all magic cookie

[ec2-user@ipaddress ~]$ xauth list
ipaddress/unix:12 MIT-MAGIC-COOKIE-1 7e53e7600ff4177d7bbc66bde0a1b1ca
ipaddress/unix:11 MIT-MAGIC-COOKIE-1 e3d1a8915484c929ef3e809b047e6352
ipaddress/unix:10 MIT-MAGIC-COOKIE-1 07b3de3093cef835c19239ea952231b7

4) Show DISPLAY variable

[ec2-user@ipaddress ~]$ env|grep DISPLAY
DISPLAY=localhost:10.0

5) Create /tmp/xauth based on current DISPLAY variable

[ec2-user@ipaddress ~]$ xauth list | grep unix`echo $DISPLAY | cut -c10-12` > /tmp/xauth
[ec2-user@ipaddress ~]$ ll /tmp/xauth ; cat /tmp/xauth 
-rw-rw-r-- 1 ec2-user ec2-user 78 Dec 7 14:47 /tmp/xauth
ipaddress/unix:10 MIT-MAGIC-COOKIE-1 07b3de3093cef835c19239ea952231b7

6) Sudo to oracle

[ec2-user@ipaddress ~]$ sudo su - oracle
Last login: Fri Dec 7 14:43:12 UTC 2018 on pts/0

7) Add and Verify xauth

[oracle@ipaddress ~]$ xauth add `cat /tmp/xauth`
[oracle@ipaddress ~]$ xauth list
ipaddress/unix:10 MIT-MAGIC-COOKIE-1 07b3de3093cef835c19239ea952231b7

8) Verify and Add DISPLAY variable

[oracle@ipaddress ~]$ env|grep DISPLAY
[oracle@ipaddress ~]$ export DISPLAY=localhost:10.0

9) Test xclock works from oracle

[oracle@ipaddress ~]$ xclock
Warning: Missing charsets in String to FontSet conversion
^C
[oracle@ipaddress ~]$

Now you should be able to see the clock and so other graphical resources, like DBCA and so on.

Hope it helps!

Database, Enterprise Manager

OMSPatcher finds that previous patching session is not yet completed – What to do?

 , ,

Hey all,
As usual, a client reached out with this issue:

OMSPatcher finds that previous patching session is not yet completed.
Please refer log file "/u01/app/oracle/middleware/cfgtoollogs/omspatcher/28018178/omspatcher_2018-07-09_23-44-58PM_deploy.log" 
for the previous session and execute the script "/u01/app/oracle/middleware/.omspatcher_storage/oms_session/scripts_2018-07-09_23-44-39PM/run_script_singleoms_resume.sh"  to complete the previous session. OMSPatcher can proceed to execute new operations only if previous session is completed successfully.

Interesting, right?
This means a patch execution in July failed and it wasn’t noticed.

What to do? Point is, the error itself already say what needs to be done.
You just may want to make it properly. How? Here is a quick Action Plan:

ZER0) Check the Deploy log to understand the root cause for the failure on previous patch and fix it.

In my case?
Not all required components were down.
A simple “stop oms” stops only the OMS managed server, JVMD engine, and HTTP server but leaves Node Manager and Administration Server running.
However, a “stop oms -all” stops all Enterprise Manager processes including Administration Server, OMS, HTTP Server, Node Manager, Management Server, JVMD engine, and Oracle BI Publisher (if it is configured on the host). This was the fixing.

Step-by-Step:

1. Blackout targets to avoid unwanted pages.
– On OEM: Enterprise–>Monitoring–>Blackouts

2. Shutdown OMS and AGENT

cd $AGENT_HOME/bin
./emctl stop agent
cd $OMS_HOME/bin
./emctl stop oms -all

3. Resume Patching with issue (with provided command)
(in my case):

/u01/app/oracle/middleware/.omspatcher_storage/oms_session/scripts_218-07-09_23-44-39PM/run_script_singleoms_resume.sh

4. Verify patches got installed

$OMS_HOME/OPatch/opatch lsinventory
$OMS_HOME/OMSPatcher/omspatcher lspatches

5. Start the OMS and agent

cd $AGENT_HOME/bin
./emctl start agent
cd $OMS_HOME/bin
./emctl start oms
./emctl status oms -details

6. Sync EMCLI with server changes:

$OMS_HOME/bin/emcli login -username=sysman
Enter password : <-- sysman password
$OMS_HOME/bin/emcli sync

More“OMSPatcher finds that previous patching session is not yet completed – What to do?”

Cloud Services | Exadata | Linux, Database, Oracle Engineered Systems | Exadata

Exadata: Generate a Sundiag

 ,

Hello all!

Today’s post is a very simple one, once seems sometimes a simple and success case post is appreciated. So, how to generate a Sundiag Report for Oracle?

First, what is a Sundiag?
It consists in an Oracle hardware healthcheck list to be run on your environment. It doesn’t give us any report, but is usually requested by Oracle on SRs related to network, disk I/O or any other possible hardware/firmware related issue.

So, how to do it? Simple like this (the path is always the same):

[root@greporacel01 ~]# cd /opt/oracle.SupportTools/
[root@greporacel01 oracle.SupportTools]# /opt/oracle.SupportTools/sundiag.sh

Oracle Exadata Database Machine - Diagnostics Collection Tool

Gathering Linux information

Skipping ILOM collection. Use the ilom or snapshot options, or login to ILOM
over the network and run Snapshot separately if necessary.

/tmp/sundiag_greporacel01_1108FMM0FF_2017_01_01_02_17
Gathering Cell information

Generating diagnostics tarball and removing temp directory

====================================================================
Done. The report files are bzip2 compressed in /tmp/sundiag_greporacel01_1108FMM0FF_2017_01_01_02_17.tar.bz2
====================================================================
[root@greporacel01 oracle.SupportTools]#

Now you just need to pick this generated file and add to your SR. Simple right?

Hope it helps!

Enterprise Manager

OEM: Quickly Ignore ORA Error on Agent Layer

 ,

Hey all,
So, I had a very specific situation to ignore an error from an agent. Turns that this seems even easier and quicker to ignore an specific error using an OEM Metric… How? Using agent parameter adrAlertLogErrorCodeExcludeRegex.

How to do it? Well, [AGENT_INST]/sysman/config/emd.properties, add a line with this parameters and the Regex to ignore the desired error or message.

To ignore all ORA-700, por example, it can be done by:

adrAlertLogErrorCodeExcludeRegex=.*700.*

Now to ignore, for example, ORA 700 [kskvmstatact: excessive swapping observed]

adrAlertLogErrorCodeExcludeRegex=.*kskvmstatact.*

After this, a restart on agent is required.

This is also well documented as per MOS EM 12c, 13c: How to Disable or Suppress OEM Alerts for Alert Log Error ORA-700 (Doc ID 2406779.1)

Hope it helps!

Cloud Services, Cloud Services | Exadata | Linux, Database, SQL | PLSQL

AWS – Kill Sessions on RDS Oracle Database

 , ,

Hi all!
Today is a quick tip. Actually, just because more than once I was about to kill a session in a RDS session and it was simply not possible with my usual ALTER SYSTEM KILL SESSION scripts… 🙂

Know how to do it? Here it goes:

begin
rdsadmin.rdsadmin_util.kill(
sid => &sid,
serial => &serial);
end;
/

Putted it in a script: @killrds.sql

Cheers!

Enterprise Manager

OEM 13C: How to Set Up Out Of Band Notifications

So, after a quiet weekend on a client, noticed I was not being paged for a reason: OMS was down! 😀

Ok, so, how to monitor the monitoring easily?
OEM 13c has a feature called Out of Band Notification, which allows configuring an agent with email credentials to send notifications when he is not able to communicate with OMS and Repo DB are down.

Details of that configuration is on this MOS note: EM 13c, 12c: How to Set Up Out Of Band Email Notification in Enterprise Manager Cloud Control (Doc ID 1472854.1)

How does it work?
The agent on the OMS host checks the status of the ‘OMS and repository’ target (oracle_emrep) by running the metric ‘Response’ which runs the perl script:

[agent_home]/plugins/oracle.sysman.emrep.agent.plugin_12.1.0.n.0/scripts/emrepresp.pl

If the oracle_emrep target is detected as down then emrepdown.pl will be called on same directory.

The emrepdown.pl uses the perl “Net::SMTP” method to send an email using the Out Of Band email information (To Email ID, Email Gateway, From Email ID) defined in the Agent’s /sysman/config/emd.properties configuration file.

Note: this method does not currently support SSL email authentication, an internal ER (Bug 18886316 “WOULD LIKE ABILITY FOR EMREPDOWN.PL TO BE ABLE TO USE SSL” ) has been raised for this.

How to set up?
1) Run the following commands which will set the email parameters in the emd.properties file.
Do this on the chained agent (ie. the agent on the same machine as the OMS which monitors the oracle_emrep target)

a) Set the agent ORACLE_HOME

$ export ORACLE_HOME=
$ export PATH= ORACLE_HOME/bin:$PATH

Example:

$ export ORACLE_HOME=/oracle/12c/12cagent/core/12.1.0.3.0
$ export PATH=$ORACLE_HOME/bin:$PATH

b) Check if any values are currently set for the Out of Band parameters

$ emctl getproperty agent -name emd_email_address
$ emctl getproperty agent -name emd_from_email_address
$ emctl getproperty agent -name emd_email_gateway

 

If the message is returned:

emd_email_address is not a valid configuration property

It means that this is not yet set up, continue to the next section.

c) Set the Out of Band parameters

emctl setproperty agent -allow_new -name emd_email_address -value [youremailaddress]
emctl setproperty agent -allow_new -name emd_from_email_address -value [senderAddress]
emctl setproperty agent -allow_new -name emd_email_gateway -value [outgoingsmtpserver]
Example:
$ emctl setproperty agent -allow_new -name emd_email_gateway -value smtp.server.hostname
$ emctl setproperty agent -allow_new -name emd_email_address -value noc@grepora.com
$ emctl setproperty agent -allow_new -name emd_from_email_address -value 13cagent@grepora.com

TIP: The value for the emd_email_gateway can be the same as is used for ‘normal’ email notifications via the OMS. This can be accessed via setup/notifications/notification methods.

If you need to use “Use Secure Connection:SSL” normally, then this means that your mail server requires SSL authentication which means that the OOB method will not be suitable. Remember: the OOB method does not support SSL email authentication at this moment in time.

2) Stop and start the agent for these parameters to take effect.

More informations like to test this configuration can be found on MOS note: EM 13c, 12c: How to Set Up Out Of Band Email Notification in Enterprise Manager Cloud Control (Doc ID 1472854.1)

Hope that helps, cheers!

Database, Enterprise Manager

OEM: The number of hanging transactions are hang_trans is %

 , ,

Hi all!
So, today is quickie one, just to make the links. Seems this message from OEM is not clear enough for some people, specially regarding non-specialists in Oracle: This means something is in lock in your database!

If this is the case, contact a DBA.

If you ARE a DBA, you may want to read this post about easy locating and solving locks: Solving Simple Locks Through @lock2s and @killlocker.

Also, if the session if from DBLink, is always useful to read this: Lock by DBLink – How to locate the remote session?

There is also some additional/specific material about some issues and bugs in this regard here: Tag: LOCK.

I hope it helps!
Cheers!

Cloud Services | Exadata | Linux, Database, Oracle Engineered Systems | Exadata

Reduce Exadata Core Count

 , , ,

Ok, so I was preparing for a DC services migration with a client and this would involve resizing the CPU count of Exadatas for better attending those services. This way, one of the steps will require reduce CPU counts in one of the sites to be aligned with the license terms.

Checking for the steps to accomplish that, I found references to change CPU and core count, but always described in the case of increasing allocation. As per 2.7 Increasing the Number of Active Cores on Database Servers. But not so much about reducing, as this seems to be unusual…

Also considering that the planned change would be within the minimum number requirement: 2.1 Restrictions for Capacity-On-Demand on Oracle Exadata Database Machine.

Reviewing on MOS, we found the When Attempting to Change the Number of Cores, Errors With: DBM-10004 – Decreasing the Number of Active Cores is not Supported ( Doc ID 2177634.1 ), pointing to use the clause “FORCE” on “ALTER DBSERVER pendingCoreCount =x” command.

And this worked. I just disabled the iaasMode to play safe. Have a look:

[root@grepora01~]# dbmcli
DBMCLI: Release  - Production on Mon Jan 05 01:10:12 EEST 2019
Copyright (c) 2007, 2014, Oracle.  All rights reserved.
DBMCLI> LIST DBSERVER attributes coreCount
	 36/44
DBMCLI> ALTER DBSERVER pendingCoreCount = 24 force
DBM-10022: At least 26 physical cores need to be active in order to support IaaS.
DBMCLI> ALTER DBSERVER iaasMode = "off"
DBServer exadb01 successfully altered
DBMCLI> ALTER DBSERVER pendingCoreCount = 24 force
DBServer grepora01 successfully altered. Please reboot the system to make the new pendingCoreCount effective.
DBMCLI> LIST DBSERVER attributes pendingCoreCount
24/44

–> Restart the server
After restarting, it should look like:

DBMCLI> LIST DBSERVER attributes coreCount
	 24/44
DBMCLI> LIST DBSERVER attributes pendingCoreCount

Hope this helps!