Exadata: 7 Useful Commands to check Port/Sensor Alarms

Hello all!

This days I had an alarm with message below:

Message=The aggregate sensor /SYS/CABLE_CONN_STAT has a fault.

There is some useful commands I used to verify all ports/sensors in my exadata cluster.

In summary, these commands:
1) Use Intelligent Platform Management Interface (IPMI) to read the Sensor Data Record (SDR) repository
2) Use Intelligent Platform Management Interface (IPMI) to view the ILOM SP System Event Log (SEL)
3) Display all host nodes with ibhosts
4) Use ibcheckstate to scan InfiniBand fabric and validate the port logical and physical state
5) Use ibcheckerrors to scan InfiniBand fabric and validate the connectivity as described in the topology file
6) Checking for sensor healthy from switch
7) Check the overall health of the InfiniBand switch, on the Exadata switch itself

The Commands are:


1) ipmitool sdr
2) ipmitool sel list | tail -60
3) ibhosts
4) ibcheckstate -v
5) ibcheckerrors
6) showunhealthy
7)env_test

Below the complete list with commands executions and example outputs.

1) Use Intelligent Platform Management Interface (IPMI) to read the Sensor Data Record (SDR) repository:

[root@grepora03db02 ~]# ipmitool sdr
ACPI             | 0x01              | ok
HDD0/PRSNT       | 0x02              | ok
HDD0/STATE       | 0x04              | ok
HDD1/PRSNT       | 0x02              | ok
HDD1/STATE       | 0x04              | ok
HDD2/PRSNT       | 0x02              | ok
HDD2/STATE       | 0x04              | ok
HDD3/PRSNT       | 0x02              | ok
HDD3/STATE       | 0x04              | ok
HDD4/PRSNT       | 0x01              | ok
HDD4/STATE       | 0x00              | ok
HDD5/PRSNT       | 0x01              | ok
HDD5/STATE       | 0x00              | ok
HDD6/PRSNT       | 0x01              | ok
HDD6/STATE       | 0x00              | ok
HDD7/PRSNT       | 0x01              | ok
HDD7/STATE       | 0x00              | ok
NVME0/PRSNT      | 0x01              | ok
NVME1/PRSNT      | 0x01              | ok
NVME2/PRSNT      | 0x01              | ok
NVME3/PRSNT      | 0x01              | ok
HOST_ERR         | 0x01              | ok
INTSW            | 0x01              | ok
FM0/F0/TACH      | 4800 RPM          | ok
FM0/F1/TACH      | 4300 RPM          | ok
FM0/F2/TACH      | 8300 RPM          | ok
FM0/F3/TACH      | 6900 RPM          | ok
FM0/PRSNT        | 0x02              | ok
FM1/F0/TACH      | 8100 RPM          | ok
FM1/F1/TACH      | 7300 RPM          | ok
FM1/F2/TACH      | 3900 RPM          | ok
FM1/F3/TACH      | 3600 RPM          | ok
FM1/PRSNT        | 0x02              | ok
FM2/F0/TACH      | 4200 RPM          | ok
FM2/F1/TACH      | 3700 RPM          | ok
FM2/F2/TACH      | 6900 RPM          | ok
FM2/F3/TACH      | 6100 RPM          | ok
FM2/PRSNT        | 0x02              | ok
FM3/F0/TACH      | 6900 RPM          | ok
FM3/F1/TACH      | 5700 RPM          | ok
FM3/F2/TACH      | 4600 RPM          | ok
FM3/F3/TACH      | 4000 RPM          | ok
FM3/PRSNT        | 0x02              | ok
P0/D0/PRSNT      | 0x02              | ok
P0/D1/PRSNT      | 0x02              | ok
P0/D2/PRSNT      | 0x01              | ok
P0/D3/PRSNT      | 0x02              | ok
P0/D4/PRSNT      | 0x02              | ok
P0/D5/PRSNT      | 0x01              | ok
P0/D6/PRSNT      | 0x01              | ok
P0/D7/PRSNT      | 0x02              | ok
P0/D8/PRSNT      | 0x02              | ok
P0/D9/PRSNT      | 0x01              | ok
P0/D10/PRSNT     | 0x02              | ok
P0/D11/PRSNT     | 0x02              | ok
P0/PRSNT         | 0x01              | ok
P0/V_DIMM        | 1.23 Volts        | ok
P1/D0/PRSNT      | 0x02              | ok
P1/D1/PRSNT      | 0x02              | ok
P1/D2/PRSNT      | 0x01              | ok
P1/D3/PRSNT      | 0x02              | ok
P1/D4/PRSNT      | 0x02              | ok
P1/D5/PRSNT      | 0x01              | ok
P1/D6/PRSNT      | 0x01              | ok
P1/D7/PRSNT      | 0x02              | ok
P1/D8/PRSNT      | 0x02              | ok
P1/D9/PRSNT      | 0x01              | ok
P1/D10/PRSNT     | 0x02              | ok
P1/D11/PRSNT     | 0x02              | ok
P1/PRSNT         | 0x01              | ok
P1/V_DIMM        | 1.24 Volts        | ok
R1/PCIE1/PRSNT   | 0x01              | ok
MB/RISER1/PRSNT  | 0x02              | ok
R2/PCIE2/PRSNT   | 0x02              | ok
MB/RISER2/PRSNT  | 0x02              | ok
R3/PCIE3/PRSNT   | 0x02              | ok
R3/PCIE4/PRSNT   | 0x02              | ok
MB/RISER3/PRSNT  | 0x02              | ok
T_CORE_NET01     | 70 degrees C      | ok
T_CORE_NET23     | 71 degrees C      | ok
T_IN_PS          | 31 degrees C      | ok
T_IN_SLOT1       | 43 degrees C      | ok
T_IN_SLOT2       | 50 degrees C      | ok
T_IN_SLOT3       | 38 degrees C      | ok
T_OUT_SLOT1      | 49 degrees C      | ok
T_OUT_SLOT2      | 50 degrees C      | ok
T_OUT_SLOT3      | 48 degrees C      | ok
PS0/PRSNT        | 0x02              | ok
PS0/P_IN         | 180 Watts         | ok
PS0/P_OUT        | 170 Watts         | ok
PS0/STATE        | 0x01              | ok
PS0/T_OUT        | 36 degrees C      | ok
PS0/V_12V        | 12 Volts          | ok
PS0/V_12V_STBY   | 12 Volts          | ok
PS0/V_IN         | 206 Volts         | ok
PS1/PRSNT        | 0x02              | ok
PS1/P_IN         | 180 Watts         | ok
PS1/P_OUT        | 160 Watts         | ok
PS1/STATE        | 0x01              | ok
PS1/T_OUT        | 38 degrees C      | ok
PS1/V_12V        | 12 Volts          | ok
PS1/V_12V_STBY   | 12 Volts          | ok
PS1/V_IN         | 204 Volts         | ok
PWRBS            | no reading        | ns
T_AMB            | 16 degrees C      | ok
/SYS/VPS         | 360 Watts         | ok
VPS_CPUS         | 180 Watts         | ok
VPS_FANS         | 10 Watts          | ok
VPS_MEMORY       | 45 Watts          | ok

 

2) Use Intelligent Platform Management Interface (IPMI) to view the ILOM SP System Event Log (SEL):

[root@grepora03db02 ~]# ipmitool sel list | tail -60
 20d | 09/14/2017 | 20:51:01 | System ACPI Power State #0x26 | S0/G0: working | Deasserted
 20e | 09/14/2017 | 20:51:01 | System ACPI Power State #0x26 | S5/G2: soft-off | Asserted
 20f | 09/14/2017 | 20:53:57 | System Boot Initiated | System Restart | Asserted
 210 | 09/14/2017 | 20:53:57 | System Firmware Progress | Management controller initialization | Asserted
 211 | 09/14/2017 | 20:53:58 | System Firmware Progress | SMBus initialization | Asserted
 212 | 09/14/2017 | 20:53:59 | System Firmware Progress | Primary CPU initialization | Asserted
 213 | 09/14/2017 | 20:54:00 | System Firmware Progress | Memory initialization | Asserted
 214 | 09/14/2017 | 20:54:00 | System Boot Initiated | Initiated by warm reset | Asserted
 215 | 09/14/2017 | 20:54:00 | System Firmware Progress | Management controller initialization | Asserted
 216 | 09/14/2017 | 20:54:01 | System Firmware Progress | SMBus initialization | Asserted
 217 | 09/14/2017 | 20:54:02 | System Firmware Progress | Primary CPU initialization | Asserted
 218 | 09/14/2017 | 20:54:02 | System Firmware Progress | Memory initialization | Asserted
 219 | 09/14/2017 | 20:54:04 | System ACPI Power State #0x26 | S0/G0: working | Asserted
 21a | 09/14/2017 | 20:54:04 | System ACPI Power State #0x26 | S5/G2: soft-off | Deasserted
 21b | 09/14/2017 | 20:54:43 | System Firmware Progress | Cache initialization | Asserted
 21c | 09/14/2017 | 20:54:46 | System Firmware Progress | Secondary CPU Initialization | Asserted
 21d | 09/14/2017 | 20:55:01 | System Firmware Progress | PCI resource configuration | Asserted
 21e | 09/14/2017 | 20:55:06 | System Firmware Progress | PCI resource configuration | Asserted
 21f | 09/14/2017 | 20:55:15 | System ACPI Power State #0x26 | S0/G0: working | Deasserted
 220 | 09/14/2017 | 20:55:15 | System ACPI Power State #0x26 | S5/G2: soft-off | Asserted
 221 | 09/14/2017 | 20:55:19 | System Boot Initiated | System Restart | Asserted
 222 | 09/14/2017 | 20:55:19 | System Firmware Progress | Management controller initialization | Asserted
 223 | 09/14/2017 | 20:55:20 | System Firmware Progress | SMBus initialization | Asserted
 224 | 09/14/2017 | 20:55:21 | System Firmware Progress | Primary CPU initialization | Asserted
 225 | 09/14/2017 | 20:55:21 | System Firmware Progress | Memory initialization | Asserted
 226 | 09/14/2017 | 20:55:22 | System Boot Initiated | Initiated by warm reset | Asserted
 227 | 09/14/2017 | 20:55:22 | System Firmware Progress | Management controller initialization | Asserted
 228 | 09/14/2017 | 20:55:22 | System Firmware Progress | SMBus initialization | Asserted
 229 | 09/14/2017 | 20:55:24 | System Firmware Progress | Primary CPU initialization | Asserted
 22a | 09/14/2017 | 20:55:24 | System Firmware Progress | Memory initialization | Asserted
 22b | 09/14/2017 | 20:55:30 | System ACPI Power State #0x26 | S0/G0: working | Asserted
 22c | 09/14/2017 | 20:55:30 | System ACPI Power State #0x26 | S5/G2: soft-off | Deasserted
 22d | 09/14/2017 | 20:56:04 | System Firmware Progress | Cache initialization | Asserted
 22e | 09/14/2017 | 20:56:08 | System Firmware Progress | Secondary CPU Initialization | Asserted
 22f | 09/14/2017 | 20:56:23 | System Firmware Progress | PCI resource configuration | Asserted
 230 | 09/14/2017 | 20:56:28 | System Firmware Progress | PCI resource configuration | Asserted
 231 | 09/14/2017 | 20:56:35 | System Firmware Progress | Video initialization | Asserted
 232 | 09/14/2017 | 20:56:35 | System Firmware Progress | Option ROM initialization | Asserted
 233 | 09/14/2017 | 20:56:38 | System Firmware Progress | Keyboard controller initialization | Asserted
 234 | 09/14/2017 | 20:56:41 | System Firmware Progress | Option ROM initialization | Asserted
 235 | 09/14/2017 | 20:57:20 | System Firmware Progress | Hard-disk initialization | Asserted
 236 | 09/14/2017 | 20:57:20 | System Firmware Progress | Option ROM initialization | Asserted
 237 | 09/14/2017 | 20:57:29 | System Firmware Progress | System boot initiated | Asserted
 238 | 09/14/2017 | 20:57:29 | System Firmware Progress | System boot initiated | Asserted
 239 | 09/14/2017 | 20:59:13 | System Firmware Progress | Management controller initialization | Asserted
 23a | 09/14/2017 | 20:59:13 | System Firmware Progress | SMBus initialization | Asserted
 23b | 09/14/2017 | 20:59:14 | System Firmware Progress | Primary CPU initialization | Asserted
 23c | 09/14/2017 | 20:59:14 | System Firmware Progress | Memory initialization | Asserted
 23d | 09/14/2017 | 20:59:55 | System Firmware Progress | Cache initialization | Asserted
 23e | 09/14/2017 | 20:59:58 | System Firmware Progress | Secondary CPU Initialization | Asserted
 23f | 09/14/2017 | 21:00:13 | System Firmware Progress | PCI resource configuration | Asserted
 240 | 09/14/2017 | 21:00:18 | System Firmware Progress | PCI resource configuration | Asserted
 241 | 09/14/2017 | 21:00:25 | System Firmware Progress | Video initialization | Asserted
 242 | 09/14/2017 | 21:00:25 | System Firmware Progress | Option ROM initialization | Asserted
 243 | 09/14/2017 | 21:00:28 | System Firmware Progress | Keyboard controller initialization | Asserted
 244 | 09/14/2017 | 21:00:32 | System Firmware Progress | Option ROM initialization | Asserted
 245 | 09/14/2017 | 21:01:10 | System Firmware Progress | Hard-disk initialization | Asserted
 246 | 09/14/2017 | 21:01:10 | System Firmware Progress | Option ROM initialization | Asserted
 247 | 09/14/2017 | 21:01:18 | System Firmware Progress | System boot initiated | Asserted
 248 | 09/14/2017 | 21:01:18 | System Firmware Progress | System boot initiated | Asserted

 

3) Display all host nodes with ibhosts:

[root@grepora03db02 ~]# ibhosts
Ca	: 0x0021280001cf2dea ports 2 "grepora01nas01 PCIe 1"
Ca	: 0x0021280001cf7d6a ports 2 "grepora01db04 S 10.10.10.7 HCA-1"
Ca	: 0x0021280001a13cbc ports 2 "grepora01db02 S 10.10.10.2 HCA-1"
Ca	: 0x0021280001cf798e ports 2 "grepora01db03 S 10.10.10.6 HCA-1"
Ca	: 0x0021280001a0b038 ports 2 "grepora01cel07 C 10.10.10.11 HCA-1"
Ca	: 0x0021280001a135d0 ports 2 "grepora01db01 S 10.10.10.1 HCA-1"
Ca	: 0x0021280001cedbca ports 2 "grepora01cel05 C 10.10.10.9 HCA-1"
Ca	: 0x0021280001cf6006 ports 2 "grepora01cel06 C 10.10.10.10 HCA-1"
Ca	: 0x0021280001a151c0 ports 2 "grepora01cel03 C 10.10.10.5 HCA-1"
Ca	: 0x0021280001a16a12 ports 2 "grepora01cel04 C 10.10.10.8 HCA-1"
Ca	: 0x0021280001a15364 ports 2 "grepora01cel01 C 10.10.10.3 HCA-1"
Ca	: 0x0021280001a1590a ports 2 "grepora01cel02 C 10.10.10.4 HCA-1"
Ca	: 0x0010e00001333318 ports 2 "grepora01nas03 PCIe 6"
Ca	: 0x0010e00001333704 ports 2 "grepora01nas03 PCIe 5"
Ca	: 0x0010e00001330f3c ports 2 "grepora01nas04 PCIe 6"
Ca	: 0x0010e00001332f30 ports 2 "grepora01nas04 PCIe 5"
Ca	: 0x0010e0000128e3e4 ports 2 "grepora02db02 S 10.10.10.24 HCA-1"
Ca	: 0x0010e0000128e18c ports 2 "grepora02db01 S 10.10.10.28 HCA-1"
Ca	: 0x0010e00001289c6c ports 2 "grepora02cel03 C 10.10.10.27 HCA-1"
Ca	: 0x0010e0000128bea4 ports 2 "grepora02cel01 C 10.10.10.25 HCA-1"
Ca	: 0x0010e00001289d90 ports 2 "grepora02cel02 C 10.10.10.26 HCA-1"
Ca	: 0x0010e00001868f40 ports 2 "grepora03db04 S 10.10.10.35,10.10.10.36 HCA-1"
Ca	: 0x0010e00001859cd0 ports 2 "grepora03db03 S 10.10.10.33,10.10.10.34 HCA-1"
Ca	: 0x0010e000018640a0 ports 2 "grepora03db01 S 10.10.10.29,10.10.10.30 HCA-1"
Ca	: 0x0010e00001887928 ports 2 "grepora03cel07 C 10.10.10.49,10.10.10.50 HCA-1"
Ca	: 0x0010e0000185cd00 ports 2 "grepora03cel05 C 10.10.10.45,10.10.10.46 HCA-1"
Ca	: 0x0010e00001868e80 ports 2 "grepora03cel06 C 10.10.10.47,10.10.10.48 HCA-1"
Ca	: 0x0010e0000185e5a0 ports 2 "grepora03cel04 C 10.10.10.43,10.10.10.44 HCA-1"
Ca	: 0x0010e0000185e5f0 ports 2 "grepora03cel03 C 10.10.10.41,10.10.10.42 HCA-1"
Ca	: 0x0010e00001638be4 ports 2 "grepora03cel01 C 10.10.10.37,10.10.10.38 HCA-1"
Ca	: 0x0010e0000187c658 ports 2 "grepora03cel02 C 10.10.10.39,10.10.10.40 HCA-1"
Ca	: 0x0010e0000185c130 ports 2 "grepora03db02 S 10.10.10.31,10.10.10.32 HCA-1"

 

4) Use ibcheckstate to scan InfiniBand fabric and validate the port logical and physical state:
It reports any logical port with state other than active or physical with state other than LinkUp.

[root@grepora03db02 ~]# ibcheckstate -v

# Checking Switch: nodeguid 0x0021284692dea0a0
Node check lid 3:  OK 
Port check lid 3 port 1:  OK 
Port check lid 3 port 2:  OK 
Port check lid 3 port 3:  OK 
Port check lid 3 port 4:  OK 
Port check lid 3 port 5:  OK 
Port check lid 3 port 6:  OK 

Port check lid 3 port 7:  OK 
Port check lid 3 port 8:  OK 
Port check lid 3 port 9:  OK 
Port check lid 3 port 10:  OK 
Port check lid 3 port 12:  OK 
Port check lid 3 port 13:  OK 
Port check lid 3 port 14:  OK 
Port check lid 3 port 15:  OK 
Port check lid 3 port 16:  OK 
Port check lid 3 port 17:  OK 
Port check lid 3 port 18:  OK 
Port check lid 3 port 29:  OK 
Port check lid 3 port 31:  OK 
Port check lid 3 port 32:  OK 

# Checking Switch: nodeguid 0x0021284692d1a0a0
Node check lid 1:  OK 
Port check lid 1 port 1:  OK 
Port check lid 1 port 2:  OK 
Port check lid 1 port 3:  OK 
Port check lid 1 port 4:  OK 
Port check lid 1 port 5:  OK 
Port check lid 1 port 6:  OK 
Port check lid 1 port 7:  OK 
Port check lid 1 port 8:  OK 
Port check lid 1 port 9:  OK 
Port check lid 1 port 10:  OK 
Port check lid 1 port 12:  OK 
Port check lid 1 port 13:  OK 
Port check lid 1 port 14:  OK 
Port check lid 1 port 15:  OK 
Port check lid 1 port 16:  OK 
Port check lid 1 port 17:  OK 
Port check lid 1 port 18:  OK 
Port check lid 1 port 29:  OK 
Port check lid 1 port 31:  OK 
Port check lid 1 port 32:  OK 

# Checking Switch: nodeguid 0x0010e035c814a0a0
Node check lid 7:  OK 
Port check lid 7 port 1:  OK 
Port check lid 7 port 2:  OK 
Port check lid 7 port 4:  OK 
Port check lid 7 port 7:  OK 
Port check lid 7 port 10:  OK 
Port check lid 7 port 11:  OK 
Port check lid 7 port 13:  OK 
Port check lid 7 port 14:  OK 
Port check lid 7 port 15:  OK 
Port check lid 7 port 16:  OK 
Port check lid 7 port 17:  OK 
Port check lid 7 port 18:  OK 
Port check lid 7 port 27:  OK 
Port check lid 7 port 29:  OK 
Port check lid 7 port 30:  OK 
Port check lid 7 port 31:  OK 
Port check lid 7 port 33:  OK 
Port check lid 7 port 34:  OK 
Port check lid 7 port 35:  OK 
Port check lid 7 port 36:  OK 

# Checking Switch: nodeguid 0x0010e035bed4a0a0
Node check lid 27:  OK 
Port check lid 27 port 1:  OK 
Port check lid 27 port 2:  OK 
Port check lid 27 port 4:  OK 
Port check lid 27 port 7:  OK 
Port check lid 27 port 10:  OK 
Port check lid 27 port 11:  OK 
Port check lid 27 port 13:  OK 
Port check lid 27 port 14:  OK 
Port check lid 27 port 15:  OK 
Port check lid 27 port 16:  OK 
Port check lid 27 port 17:  OK 
Port check lid 27 port 18:  OK 
Port check lid 27 port 27:  OK 
Port check lid 27 port 29:  OK 
Port check lid 27 port 30:  OK 
Port check lid 27 port 31:  OK 
Port check lid 27 port 33:  OK 
Port check lid 27 port 34:  OK 
Port check lid 27 port 35:  OK 
Port check lid 27 port 36:  OK 

# Checking Switch: nodeguid 0x0010e0801a0aa0a0
Node check lid 62:  OK 
Port check lid 62 port 1:  OK 
Port check lid 62 port 2:  OK 
Port check lid 62 port 3:  OK 
Port check lid 62 port 4:  OK 
Port check lid 62 port 5:  OK 
Port check lid 62 port 6:  OK 
Port check lid 62 port 7:  OK 
Port check lid 62 port 8:  OK 
Port check lid 62 port 9:  OK 
Port check lid 62 port 10:  OK 
Port check lid 62 port 12:  OK 
Port check lid 62 port 13:  OK 
Port check lid 62 port 14:  OK 
Port check lid 62 port 15:  OK 
Port check lid 62 port 16:  OK 
Port check lid 62 port 17:  OK 
Port check lid 62 port 18:  OK 
Port check lid 62 port 31:  OK 
Port check lid 62 port 32:  OK 

# Checking Switch: nodeguid 0x00212846965fa0a0
Node check lid 80:  OK 
Port check lid 80 port 13:  OK 
Port check lid 80 port 14:  OK 
Port check lid 80 port 15:  OK 
Port check lid 80 port 16:  OK 
Port check lid 80 port 19:  OK 
Port check lid 80 port 20:  OK 
Port check lid 80 port 21:  OK 
Port check lid 80 port 22:  OK 
Port check lid 80 port 23:  OK 
Port check lid 80 port 24:  OK 
Port check lid 80 port 25:  OK 
Port check lid 80 port 26:  OK 
Port check lid 80 port 27:  OK 
Port check lid 80 port 28:  OK 
Port check lid 80 port 29:  OK 
Port check lid 80 port 30:  OK 
Port check lid 80 port 31:  OK 
Port check lid 80 port 32:  OK 
Port check lid 80 port 33:  OK 
Port check lid 80 port 34:  OK 

# Checking Switch: nodeguid 0x0010e0801ff2a0a0
Node check lid 54:  OK 
Port check lid 54 port 7:  OK 
Port check lid 54 port 8:  OK 
Port check lid 54 port 9:  OK 
Port check lid 54 port 10:  OK 
Port check lid 54 port 11:  OK 
Port check lid 54 port 12:  OK 
Port check lid 54 port 17:  OK 
Port check lid 54 port 18:  OK 
Port check lid 54 port 19:  OK 
Port check lid 54 port 20:  OK 
Port check lid 54 port 21:  OK 
Port check lid 54 port 22:  OK 
Port check lid 54 port 25:  OK 
Port check lid 54 port 26:  OK 
Port check lid 54 port 27:  OK 
Port check lid 54 port 28:  OK 
Port check lid 54 port 29:  OK 
Port check lid 54 port 30:  OK 
Port check lid 54 port 35:  OK 
Port check lid 54 port 36:  OK 

# Checking Switch: nodeguid 0x0010e0801a9da0a0
Node check lid 53:  OK 
Port check lid 53 port 1:  OK 
Port check lid 53 port 2:  OK 
Port check lid 53 port 3:  OK 
Port check lid 53 port 4:  OK 
Port check lid 53 port 5:  OK 
Port check lid 53 port 6:  OK 
Port check lid 53 port 7:  OK 
Port check lid 53 port 8:  OK 
Port check lid 53 port 9:  OK 
Port check lid 53 port 10:  OK 
Port check lid 53 port 12:  OK 
Port check lid 53 port 13:  OK 
Port check lid 53 port 14:  OK 
Port check lid 53 port 15:  OK 
Port check lid 53 port 16:  OK 
Port check lid 53 port 17:  OK 
Port check lid 53 port 18:  OK 
Port check lid 53 port 31:  OK 
Port check lid 53 port 32:  OK 

# Checking Ca: nodeguid 0x0021280001cf2dea
Node check lid 13:  OK 
Port check lid 13 port 1:  OK 
Port check lid 13 port 2:  OK 

# Checking Ca: nodeguid 0x0021280001cf7d6a
Node check lid 25:  OK 
Port check lid 25 port 1:  OK 
Port check lid 25 port 2:  OK 

# Checking Ca: nodeguid 0x0021280001a13cbc
Node check lid 5:  OK 
Port check lid 5 port 1:  OK 
Port check lid 5 port 2:  OK 

# Checking Ca: nodeguid 0x0021280001cf798e
Node check lid 23:  OK 
Port check lid 23 port 1:  OK 
Port check lid 23 port 2:  OK 

# Checking Ca: nodeguid 0x0021280001a0b038
Node check lid 21:  OK 
Port check lid 21 port 1:  OK 
Port check lid 21 port 2:  OK 

# Checking Ca: nodeguid 0x0021280001a135d0
Node check lid 2:  OK 
Port check lid 2 port 1:  OK 
Port check lid 2 port 2:  OK 

# Checking Ca: nodeguid 0x0021280001cedbca
Node check lid 17:  OK 
Port check lid 17 port 1:  OK 
Port check lid 17 port 2:  OK 

# Checking Ca: nodeguid 0x0021280001cf6006
Node check lid 19:  OK 
Port check lid 19 port 1:  OK 
Port check lid 19 port 2:  OK 

# Checking Ca: nodeguid 0x0021280001a151c0
Node check lid 11:  OK 
Port check lid 11 port 1:  OK 
Port check lid 11 port 2:  OK 

# Checking Ca: nodeguid 0x0021280001a16a12
Node check lid 15:  OK 
Port check lid 15 port 1:  OK 
Port check lid 15 port 2:  OK 

# Checking Ca: nodeguid 0x0021280001a15364
Node check lid 38:  OK 
Port check lid 38 port 1:  OK 
Port check lid 38 port 2:  OK 

# Checking Ca: nodeguid 0x0021280001a1590a
Node check lid 9:  OK 
Port check lid 9 port 1:  OK 
Port check lid 9 port 2:  OK 

# Checking Ca: nodeguid 0x0010e00001333704
Node check lid 42:  OK 
Port check lid 42 port 1:  OK 
Port check lid 42 port 2:  OK 

# Checking Ca: nodeguid 0x0010e00001333318
Node check lid 41:  OK 
Port check lid 41 port 1:  OK 
Port check lid 41 port 2:  OK 

# Checking Ca: nodeguid 0x0010e00001330f3c
Node check lid 40:  OK 
Port check lid 40 port 1:  OK 
Port check lid 40 port 2:  OK 

# Checking Ca: nodeguid 0x0010e00001332f30
Node check lid 39:  OK 
Port check lid 39 port 1:  OK 
Port check lid 39 port 2:  OK 

# Checking Ca: nodeguid 0x0010e0000128e3e4
Node check lid 36:  OK 
Port check lid 36 port 1:  OK 
Port check lid 36 port 2:  OK 

# Checking Ca: nodeguid 0x0010e0000128e18c
Node check lid 34:  OK 
Port check lid 34 port 1:  OK 
Port check lid 34 port 2:  OK 

# Checking Ca: nodeguid 0x0010e00001289c6c
Node check lid 32:  OK 
Port check lid 32 port 1:  OK 
Port check lid 32 port 2:  OK 

# Checking Ca: nodeguid 0x0010e0000128bea4
Node check lid 28:  OK 
Port check lid 28 port 1:  OK 
Port check lid 28 port 2:  OK 

# Checking Ca: nodeguid 0x0010e00001289d90
Node check lid 30:  OK 
Port check lid 30 port 1:  OK 
Port check lid 30 port 2:  OK 

# Checking Ca: nodeguid 0x0010e00001868f40
Node check lid 50:  OK 
Port check lid 50 port 1:  OK 
Port check lid 50 port 2:  OK 

# Checking Ca: nodeguid 0x0010e00001859cd0
Node check lid 57:  OK 
Port check lid 57 port 1:  OK 
Port check lid 57 port 2:  OK 

# Checking Ca: nodeguid 0x0010e00001887928
Node check lid 48:  OK 
Port check lid 48 port 1:  OK 
Port check lid 48 port 2:  OK 

# Checking Ca: nodeguid 0x0010e000018640a0
Node check lid 55:  OK 
Port check lid 55 port 1:  OK 
Port check lid 55 port 2:  OK 

# Checking Ca: nodeguid 0x0010e0000185cd00
Node check lid 47:  OK 
Port check lid 47 port 1:  OK 
Port check lid 47 port 2:  OK 

# Checking Ca: nodeguid 0x0010e00001868e80
Node check lid 52:  OK 
Port check lid 52 port 1:  OK 
Port check lid 52 port 2:  OK 

# Checking Ca: nodeguid 0x0010e0000185e5a0
Node check lid 56:  OK 
Port check lid 56 port 1:  OK 
Port check lid 56 port 2:  OK 

# Checking Ca: nodeguid 0x0010e0000185e5f0
Node check lid 59:  OK 
Port check lid 59 port 1:  OK 
Port check lid 59 port 2:  OK 

# Checking Ca: nodeguid 0x0010e00001638be4
Node check lid 58:  OK 
Port check lid 58 port 1:  OK 
Port check lid 58 port 2:  OK 

# Checking Ca: nodeguid 0x0010e0000187c658
Node check lid 51:  OK 
Port check lid 51 port 1:  OK 
Port check lid 51 port 2:  OK 

# Checking Ca: nodeguid 0x0010e0000185c130
Node check lid 49:  OK 
Port check lid 49 port 1:  OK 
Port check lid 49 port 2:  OK 

## Summary: 40 nodes checked, 0 bad nodes found
##          222 ports checked, 0 ports with bad state found

 

5) Use ibcheckerrors to scan InfiniBand fabric and validate the connectivity as described in the topology file:
The ibcheckerrors command uses the topology file to scan the InfiniBand fabric and validate the connectivity as described in the topology file, and to report errors as indicated by the port counters.

[root@grepora03db02 ~]# ibcheckerrors
#warn: counter PortRcvRemotePhysicalErrors = 100 	(threshold 100) lid 1 port 255
Error check on lid 1 (SUN DCS 36P QDR grepora01sw-ib2 10.20.10.24) port all:  FAILED 
#warn: counter PortRcvRemotePhysicalErrors = 100 	(threshold 100) lid 1 port 15
Error check on lid 1 (SUN DCS 36P QDR grepora01sw-ib2 10.20.10.24) port 15:  FAILED 
#warn: counter LinkErrorRecoveryCounter = 12 	(threshold 10) lid 7 port 255
Error check on lid 7 (SUN DCS 36P QDR grepora02sw-ibb0 10.20.10.50) port all:  FAILED 
#warn: counter LinkErrorRecoveryCounter = 12 	(threshold 10) lid 7 port 34
Error check on lid 7 (SUN DCS 36P QDR grepora02sw-ibb0 10.20.10.50) port 34:  FAILED 
#warn: counter SymbolErrorCounter = 156 	(threshold 10) lid 27 port 255
Error check on lid 27 (SUN DCS 36P QDR grepora02sw-iba0 10.20.10.49) port all:  FAILED 
#warn: counter SymbolErrorCounter = 156 	(threshold 10) lid 27 port 35
Error check on lid 27 (SUN DCS 36P QDR grepora02sw-iba0 10.20.10.49) port 35:  FAILED 
#warn: counter SymbolErrorCounter = 13896 	(threshold 10) lid 80 port 255
#warn: counter LinkErrorRecoveryCounter = 12 	(threshold 10) lid 80 port 255
#warn: counter PortRcvErrors = 145 	(threshold 10) lid 80 port 255
Error check on lid 80 (SUN DCS 36P QDR grepora01sw-ib1 10.20.10.23) port all:  FAILED 
#warn: counter SymbolErrorCounter = 118 	(threshold 10) lid 80 port 19
#warn: counter PortRcvErrors = 115 	(threshold 10) lid 80 port 19
Error check on lid 80 (SUN DCS 36P QDR grepora01sw-ib1 10.20.10.23) port 19:  FAILED 
#warn: counter SymbolErrorCounter = 13777 	(threshold 10) lid 80 port 22
#warn: counter LinkErrorRecoveryCounter = 12 	(threshold 10) lid 80 port 22
#warn: counter PortRcvErrors = 29 	(threshold 10) lid 80 port 22
Error check on lid 80 (SUN DCS 36P QDR grepora01sw-ib1 10.20.10.23) port 22:  FAILED 
#warn: counter PortRcvErrors = 40 	(threshold 10) lid 53 port 255
Error check on lid 53 (SUN DCS 36P QDR grepora03sw-iba01 10.20.10.76) port all:  FAILED 
#warn: counter PortRcvErrors = 38 	(threshold 10) lid 53 port 8
Error check on lid 53 (SUN DCS 36P QDR grepora03sw-iba01 10.20.10.76) port 8:  FAILED 

## Summary: 40 nodes checked, 0 bad nodes found
##          222 ports checked, 6 ports have errors beyond threshold

 

6) Checking for sensor healthy from switch:
* Running from a leaf switch.

[root@grepora03sw-ibb01 ~]# showunhealthy
OK - No unhealthy sensors

 

7) Check the overall health of the InfiniBand switch, on the Exadata switch itself:

[root@grepora03sw-ibb01 ~]# env_test
Environment test started:
Starting Environment Daemon test:
Environment daemon running
Environment Daemon test returned OK
Starting Voltage test:
Voltage ECB OK
Measured 3.3V Main = 3.25 V
Measured 3.3V Standby = 3.35 V
Measured 12V = 12.03 V
Measured 5V = 4.99 V
Measured VBAT = 3.09 V
Measured 2.5V = 2.50 V
Measured 1.8V = 1.78 V
Measured I4 1.2V = 1.21 V
Voltage test returned OK
Starting PSU test:
PSU 0 present OK
PSU 1 present OK
PSU test returned OK
Starting Temperature test:
Back temperature 29
Front temperature 31
SP temperature 48
Switch temperature 49, maxtemperature 57
Temperature test returned OK
Starting FAN test:
Fan 0 not present
Fan 1 running at rpm 11445
Fan 2 running at rpm 11445
Fan 3 running at rpm 11445
Fan 4 not present
FAN test returned OK
Starting Connector test:
Connector test returned OK
Starting Onboard ibdevice test:
Switch OK
All Internal ibdevices OK
Onboard ibdevice test returned OK
Starting SSD test:
SSD test returned OK
Environment test PASSED
[root@grepora03sw-ibb01 ~]#

Some references:
https://docs.oracle.com/cd/E19464-01/820-6850-11/IPMItool.html
https://docs.oracle.com/cd/E24707_01/html/E24528/z400000c1016683.html
http://docs.oracle.com/cd/E19654-01/820-7752-12/z400014c1567639.html
http://docs.oracle.com/cd/E19654-01/820-7751-12/z400014e1393674.html
Infiniband Switch in Exadata incorrectly reporting WARNING/FAILURE when Performing ‘showunhealthy’ (Doc ID 1578284.1)

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s