Oracle RAC Cache Fusion Deep Dive: How Blocks Travel Between Instances

Cache Fusion is the technology that allows Oracle RAC nodes to share a single consistent view of the database buffer cache across all instances. Without it, every cross-instance read or write would require a disk I/O. With it, dirty blocks travel directly from one instance’s buffer cache to another across the private interconnect. Understanding how this works is the foundation for diagnosing RAC-specific performance problems.

The Problem Cache Fusion Solves

In a traditional single-instance Oracle database, the buffer cache is authoritative. When an instance needs a block, it either finds it in cache or reads it from disk. In RAC, multiple instances have their own buffer caches, but they share the same datafiles. Without coordination, Instance A and Instance B could each have different versions of the same block in cache — a consistency nightmare.

Cache Fusion solves this by maintaining a Global Resource Directory (GRD) distributed across all instances. The GRD tracks which instance holds the most current version of every block, and in what state (shared/consistent read, exclusive/modified).

Block Transfer Mechanics

Current vs Consistent Read Transfers

There are two types of block transfers:

Current block transfer: Instance B needs to modify a block currently held exclusively (dirty) by Instance A. Instance A writes the block’s past image (PI) to disk (or defers it) and ships the current block to Instance B via the interconnect. Instance B can now modify it.

Consistent read (CR) block transfer: Instance B needs a read-consistent version of a block currently being modified by Instance A. Instance A constructs a CR copy (applying undo to produce the correct read-consistent image) and ships it to Instance B. Instance A retains the current block.

Global Cache Services

Two background processes manage Cache Fusion:

• LMS (Global Cache Service): Handles block-shipping requests. Multiple LMS processes run per instance (LMS0, LMS1, LMS2…) — the number scales with interconnect bandwidth and workload.
• LMD (Global Enqueue Service Daemon): Manages global enqueues (locks) coordinating access to shared structures.

# Verify LMS processes are running (on a RAC node)
ps -ef | grep lms
oracle   1234      1  0 08:00 ?  00:00:45 ora_lms0_ORCL1
oracle   1235      1  0 08:00 ?  00:00:38 ora_lms1_ORCL1

Interconnect: The Performance Critical Path

The private interconnect is the physical medium over which Cache Fusion block transfers occur. Its latency and bandwidth directly determine the overhead of cross-instance operations.

Interconnect Requirements

• Latency: < 1ms is the standard target for a well-performing RAC cluster. Latency > 2ms will cause elevated gc cr block 2-way, gc current block 2-way wait times.
• Bandwidth: Sized by peak block transfer rate. A high-throughput OLAP workload with many full table scans should use 25 GbE or InfiniBand minimum.

Diagnosing Interconnect Performance

-- Check current interconnect name and configuration
SELECT name, ip_address, is_public
FROM v$cluster_interconnects;

-- Measure actual interconnect throughput and latency (AWR-based)
SELECT stat_name,
       ROUND(value / 1024 / 1024, 2) AS mb_or_ms
FROM v$sysstat
WHERE stat_name IN (
    'gc cr blocks received',
    'gc current blocks received',
    'gc cr block receive time',
    'gc current block receive time',
    'gcs messages sent',
    'ges messages sent'
)
ORDER BY stat_name;

Compute average block transfer time:

-- Average Global Cache receive latency (in milliseconds)
SELECT ROUND(
         (SUM(CASE WHEN stat_name = 'gc cr block receive time'       THEN value ELSE 0 END) +
          SUM(CASE WHEN stat_name = 'gc current block receive time'  THEN value ELSE 0 END))
         /
         NULLIF(
           SUM(CASE WHEN stat_name = 'gc cr blocks received'         THEN value ELSE 0 END) +
           SUM(CASE WHEN stat_name = 'gc current blocks received'    THEN value ELSE 0 END)
         , 0)
         / 100   -- convert centiseconds to ms
       , 2) AS avg_gc_latency_ms
FROM v$sysstat
WHERE stat_name IN (
    'gc cr blocks received',
    'gc current blocks received',
    'gc cr block receive time',
    'gc current block receive time'
);

A result under 1 ms is healthy. 2–5 ms indicates interconnect contention or misconfiguration. Above 5 ms requires investigation.

Global Cache Wait Events: Reading the Signals

The most common Global Cache wait events and their meaning:

-- Top Global Cache waits for the current instance
SELECT event,
       total_waits,
       ROUND(time_waited / 100, 2)        AS total_wait_secs,
       ROUND(average_wait / 100, 4)       AS avg_wait_secs,
       ROUND(time_waited / total_waits / 100, 4) AS avg_wait_ms
FROM v$system_event
WHERE event LIKE 'gc%'
  AND total_waits > 0
ORDER BY time_waited DESC
FETCH FIRST 15 ROWS ONLY;

Dynamic Resource Mastering (DRM)

Oracle RAC uses DRM to move the “master” of a resource (block range) closer to the instance that accesses it most. This reduces 3-way transfers and concentrates transfers to 2-way. DRM is automatic and transparent.

You can observe DRM activity in the alert log. Excessive DRM remastering is sometimes a symptom of a poorly partitioned workload where no instance consistently owns a given block range — indicating an application or connection routing design problem.

-- Check current resource master distribution
SELECT gc_element#,
       master_node,
       current_node,
       lock_element_addr
FROM v$gc_element_diag
FETCH FIRST 20 ROWS ONLY;

Reducing Cache Fusion Overhead: Design Principles

1. Partition Workloads by Instance

If two instances are frequently transferring the same set of blocks (same tables), route the application so that each instance primarily accesses different tables or partition ranges. Sequence-generated primary key tables are particularly problematic — the right-hand leaf blocks of the index are always the hottest and will bounce between instances.

2. Use Reverse Key Indexes for Sequences

Reverse key indexes distribute sequential inserts across index leaf blocks, reducing the single hot-block bottleneck common with SEQUENCE.NEXTVAL:

CREATE INDEX idx_orders_id ON orders (order_id) REVERSE;

3. Hash Partitioning for Even Distribution

For large tables with heavy concurrent access from multiple instances, hash partitioning distributes rows across partitions, reducing the likelihood that all instances converge on the same data blocks:

CREATE TABLE orders (
  order_id   NUMBER,
  customer_id NUMBER,
  ...
)
PARTITION BY HASH (customer_id) PARTITIONS 16;

4. Services for Workload Routing

Use Oracle Services to direct connection types to specific instances. An OLTP service active only on Instance 1, and a reporting service active only on Instance 2, minimises cross-instance block transfers for read workloads:

# Add and start a service on specific instances
srvctl add service -d ORCL -s OLTP_SVC -r "ORCL1" -a "ORCL2"
srvctl add service -d ORCL -s REPORT_SVC -r "ORCL2" -a "ORCL1"
srvctl start service -d ORCL -s OLTP_SVC
srvctl start service -d ORCL -s REPORT_SVC

AWR RAC Report Analysis

The AWR RAC report provides a cluster-wide view and is the primary tool for diagnosing Cache Fusion issues:

-- Generate AWR RAC report for all instances
SELECT output
FROM TABLE(DBMS_WORKLOAD_REPOSITORY.AWR_GLOBAL_REPORT_HTML(
  l_dbid     => (SELECT dbid FROM v$database),
  l_inst_num => '1,2',
  l_bid      => (SELECT min_snap_id FROM dba_hist_snapshot WHERE ROWNUM = 1),
  l_eid      => (SELECT MAX(snap_id) FROM dba_hist_snapshot),
  l_options  => 0
));

In the RAC report, focus on the Global Cache Load Profile and Global Cache Efficiency Percentages. A high CR block receive rate combined with high average receive time is the canonical signal for interconnect contention.