First Commands When RAC Is Behaving Badly

When a RAC cluster is degraded, start at the CRS layer before touching the database.

# Top-level CRS health
crsctl check crs

Individual CRS components

crsctl check cssd     # cluster synchronization daemon
crsctl check crsd     # cluster ready services daemon
crsctl check evmd     # event manager daemon

Database resource status across all instances

srvctl status database -d MYDB

Full resource state table — read this carefully

crsctl stat res -t

In the crsctl stat res -t output, look for resources in INTERMEDIATE or OFFLINE state that should be ONLINE. A VIP showing INTERMEDIATE means network failover is in progress or stuck.

Diagnosing Interconnect Performance

The private interconnect is the most common source of RAC performance problems. In AWR, poor interconnect shows up as gc buffer busy acquire, gc cr request, and gc current request dominating Top Timed Events.

Verify the Interconnect Interface

# Which interface is configured for the interconnect?
oifcfg getif

Output: eth1  192.168.10.0  global  cluster_interconnect


Confirm the interface is actually on the private network

ip addr show eth1

Measure Interconnect Latency

# From node 1, ping node 2 private IP with 8KB packets (simulates block transfer)
ping -I eth1 192.168.10.102 -s 8192 -c 100

Acceptable: avg round-trip < 0.5ms


Investigate: avg round-trip > 1ms


Problem:     avg round-trip > 3ms or any packet loss

Global Cache Statistics in SQL

SELECT name, value
FROM v$sysstat
WHERE name IN (
  'gc cr blocks received',
  'gc current blocks received',
  'gc cr block receive time',
  'gc current block receive time'
);

Calculate average transfer time: gc cr block receive time / gc cr blocks received gives milliseconds per block. Under 5ms is normal. Over 15ms indicates interconnect congestion or a hot-block problem between instances.

Finding Hot Objects Driving Interconnect Traffic

SELECT o.object_name, o.object_type, o.owner,
       SUM(s.value) gc_waits
FROM v$segment_statistics s
JOIN dba_objects o ON s.obj# = o.object_id
WHERE s.statistic_name = 'gc buffer busy acquire'
  AND s.value > 0
GROUP BY o.object_name, o.object_type, o.owner
ORDER BY gc_waits DESC
FETCH FIRST 15 ROWS ONLY;

CRS Not Starting After Reboot

The diagnostic path starts with the CRS trace log:

# CRS trace log location
ls -lt /u01/app/grid/diag/crs/$(hostname)/crs/trace/

Scan recent entries for errors

grep -i "error|fatal|fail"   /u01/app/grid/diag/crs/$(hostname)/crs/trace/ocssd.trc | tail -50

Voting Disk Not Reachable

The most common post-reboot CRS failure is the voting disk timing out:

# Check voting disk configuration and current accessibility
crsctl query css votedisk

Output format:


0.     STATE:Online   LABEL:VOTE1   DGNAME:OCR   DEVPATH:/dev/sdc


A state of "Offline" or timeout errors means storage is the problem

Fix the storage path (SAN zoning, multipath device, NFS mount) before trying to start CRS again.

Starting CRS Manually

# OL7+ (systemd-based)
systemctl start ohas

OL6 and earlier (init.d-based)

/etc/init.d/ohasd start

Verify CRS started successfully

crsctl check crs

Disk Timeout Tuning

If your storage has higher latency (SAN over a busy fabric, or cloud block storage), the default disk timeout may be too short:

# Check current disk timeout (seconds before a node is considered dead)
crsctl get css disktimeout

Default: 200


Check misscount (seconds a node can miss a heartbeat before eviction)

crsctl get css misscount

Default: 30

Increasing these delays eviction — which can be dangerous. Change only with Oracle Support guidance.

Node Eviction — Why It Happens

Node eviction (a node forcibly rebooted by the cluster) happens because the surviving nodes cannot confirm the evicted node is alive, and a split-brain scenario would risk data corruption. It is not a crash — it is a deliberate safety mechanism.

The root cause is always in ocssd.trc on the surviving node:

grep -A 5 "evict|reconfig|EVICTION"   /u01/app/grid/diag/crs/$(hostname)/crs/trace/ocssd.trc | tail -100

Network Flap vs Storage Delay

A node that reboots itself shows Node is being evicted in its own ocssd.trc just before reboot — it could not reach the voting disk within the disk timeout and self-evicted to prevent corruption.

Resource Management and srvctl

Use srvctl for managing database instances and services in RAC. It keeps the CRS resource model consistent. Using sqlplus / as sysdba to shut down an instance directly bypasses CRS and leaves resources in a stale state.

# Start a specific instance
srvctl start instance -d MYDB -i MYDB2

Stop a specific instance cleanly

srvctl stop instance -d MYDB -i MYDB1 -o immediate

Relocate a service from MYDB1 to MYDB2

srvctl relocate service -d MYDB -s myservice -i MYDB1 -t MYDB2

Check service status

srvctl status service -d MYDB -s myservice

Stop the entire database across all nodes

srvctl stop database -d MYDB -o immediate

Use crsctl stop crs only when you need to bring down the entire CRS stack for OS maintenance — it also stops ASM, VIPs, and all cluster resources on that node.

ASM and Disk Group Issues in RAC

-- Connect to ASM instance: sqlplus / as sysasm
SELECT name, state, type,
       ROUND(total_mb/1024, 1) total_gb,
       ROUND(free_mb/1024, 1) free_gb,
       ROUND((total_mb - free_mb) * 100.0 / total_mb, 1) pct_used
FROM v$asm_diskgroup
ORDER BY name;

From the OS on any RAC node:

# List disk groups and their state
asmcmd lsdg

Check individual disk health

asmcmd lsdsk -G DATA --discovery

Adding a Disk and Rebalancing

-- Add a disk to the DATA disk group
ALTER DISKGROUP DATA ADD DISK '/dev/sde' NAME DATA_0004;

-- Rebalance at power level 4 (1=slow, 11=max speed)
ALTER DISKGROUP DATA REBALANCE POWER 4;


-- Monitor rebalance progress
SELECT group_number, operation, state, power, sofar, est_work, est_rate, est_minutes
FROM v$asm_operation
WHERE operation = 'REBAL';

Schedule rebalance for low-usage windows — it competes with I/O from all instances sharing the disk group.