Oracle RAC Interconnect Tuning: OS Configuration, NIC Bonding, and Network Validation

The private interconnect is the central nervous system of an Oracle RAC cluster. Poor interconnect performance cascades directly into gc cr block 2-way, gc current block 2-way, and related wait times. Before tuning any Oracle parameter, the interconnect hardware and OS configuration must be validated. This article covers the full stack — from NIC configuration to Oracle’s use of the network.

Interconnect Architecture Options

For new deployments, 25 GbE minimum is the practical recommendation. On Exadata X8M+, RoCE is the default interconnect technology, delivering near-InfiniBand latency over standard Ethernet infrastructure.

OS Network Configuration

Kernel Parameters for RAC Interconnect

The following /etc/sysctl.conf settings are recommended by Oracle for RAC interconnect interfaces:

# /etc/sysctl.conf - RAC interconnect tuning
net.core.rmem_default      = 262144
net.core.rmem_max          = 4194304
net.core.wmem_default      = 262144
net.core.wmem_max          = 1048576

# Increase send/receive socket buffers for high-bandwidth interconnect
net.ipv4.tcp_rmem          = 4096 87380 4194304
net.ipv4.tcp_wmem          = 4096 65536 4194304

# Reduce time-wait bucket pressure
net.ipv4.tcp_tw_reuse      = 1
net.ipv4.tcp_fin_timeout   = 30

# Increase network backlog
net.core.netdev_max_backlog = 30000

Apply without reboot:

sysctl -p /etc/sysctl.conf

NIC Ring Buffer Sizing

Increase NIC ring buffers to reduce packet drops under burst traffic:

# Check current ring buffer settings
ethtool -g bond0
# or
ethtool -g eth1

# Increase ring buffer sizes
ethtool -G eth1 rx 4096 tx 4096
ethtool -G eth2 rx 4096 tx 4096

# Persist via /etc/rc.local or systemd service

Interrupt Coalescing

For latency-sensitive RAC clusters, reduce interrupt coalescing to ensure network packets are processed immediately:

# Disable adaptive interrupt coalescing on interconnect NIC
ethtool -C eth1 adaptive-rx off adaptive-tx off rx-usecs 0 tx-usecs 0

Note: Disabling coalescing increases CPU interrupt load. On Exadata with RoCE (RDMA), this is managed by the RDMA subsystem and standard interrupt tuning does not apply.

NIC Bonding for Redundancy

Always bond two physical NICs for the private interconnect. Oracle Clusterware supports active-passive or active-active (LACP) bonding.

# Create a bond interface for the private interconnect
# /etc/sysconfig/network-scripts/ifcfg-bond0
DEVICE=bond0
TYPE=Bond
BONDING_MASTER=yes
BOOTPROTO=none
IPADDR=192.168.10.1
NETMASK=255.255.255.0
BONDING_OPTS="mode=active-backup miimon=100 downdelay=200 updelay=200"

# Slave NIC 1
# /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE=eth1
BOOTPROTO=none
MASTER=bond0
SLAVE=yes
ONBOOT=yes

# Slave NIC 2
# /etc/sysconfig/network-scripts/ifcfg-eth2
DEVICE=eth2
BOOTPROTO=none
MASTER=bond0
SLAVE=yes
ONBOOT=yes

Use active-backup mode for simplicity and predictable failover. Use 802.3ad (LACP) mode for active-active load balancing, which requires switch configuration.

Verifying Interconnect Configuration in Oracle

Clusterware View

# Verify Oracle Clusterware interconnect interface
oifcfg getif

# Output example:
# eth0  10.0.0.0  global  public
# bond0 192.168.10.0  global  cluster_interconnect

Ensure the cluster_interconnect interface maps to the bonded private NIC, not the public interface. A misconfiguration that puts RAC traffic on the public network is a common and severe installation error.

# If interconnect needs to be changed:
oifcfg setif -global bond0/192.168.10.0:cluster_interconnect

Interconnect Speed from Oracle’s Perspective

-- Oracle's view of the interconnect speed (from Cluster Health Monitor)
SELECT *
FROM v$cluster_interconnects;

NAME   IP_ADDRESS      IS_PUBLIC SOURCE
------ --------------- --------- ------
bond0  192.168.10.1    NO        Oracle Cluster Repository

Bandwidth and Latency Testing

Before installing Oracle RAC, always validate raw interconnect performance. Use the Oracle cluvfy tool and manual testing.

Using cluvfy

# Run pre-installation cluster verification
cluvfy stage -pre crsinst -n node1,node2 -verbose

Manual Bandwidth Test with iperf3

# On node2 (server)
iperf3 -s -B 192.168.10.2

# On node1 (client) — test bandwidth
iperf3 -c 192.168.10.2 -t 30 -P 4 -B 192.168.10.1

Expected results (10 GbE bonded active-passive): ~9.5 Gb/s. If you see less than 80% of theoretical bandwidth, investigate switch port configuration, duplex settings, or MTU mismatch.

Latency Test with ping

# Test interconnect round-trip latency
ping -c 100 -i 0.01 -s 1400 192.168.10.2

# Results:
# rtt min/avg/max/mdev = 0.089/0.112/0.245/0.018 ms

Latency > 1 ms on the private interconnect requires investigation before proceeding with RAC installation.

Jumbo Frames (MTU 9000)

For 10 GbE+ interconnects, enable jumbo frames to reduce per-packet overhead:

# Set MTU on interconnect interface (must match switch port MTU)
ip link set eth1 mtu 9000
ip link set eth2 mtu 9000
ip link set bond0 mtu 9000

# Persist in interface configuration
# /etc/sysconfig/network-scripts/ifcfg-bond0
MTU=9000

Verify jumbo frame support end-to-end:

# Test jumbo frame ping (no fragmentation)
ping -M do -s 8972 192.168.10.2

If this fails, the switch ports are not configured for jumbo frames — fix the switch, not the OS.

Oracle Global Cache Interconnect Efficiency Metrics

After verifying OS and hardware configuration, measure Oracle-level interconnect efficiency:

-- Global Cache Block Transfer Efficiency
SELECT name, value
FROM v$sysstat
WHERE name IN (
    'gc cr blocks received',
    'gc current blocks received',
    'gc cr block receive time',
    'gc current block receive time',
    'physical reads'
)
ORDER BY name;

Target: average CR and current block receive time (value/blocks) < 1 ms (< 100 centiseconds per block at Oracle’s internal centisecond resolution).

If the interconnect is healthy but Cache Fusion latency is still high, the problem has shifted to workload design — excessive block sharing between instances. Revisit service routing and application partitioning as described in the Cache Fusion deep dive.