Exadata X7 to X11M Migration: An In-Depth Hardware Refresh Playbook

Migrating from Exadata X7 to X11M is not a simple hardware swap — it is a generational architectural shift. The X11M introduces RoCE (RDMA over Converged Ethernet) replacing InfiniBand, PCIe Gen5 NVMe replacing earlier NVMe tiers, and a new compute architecture with AMD EPYC processors. This article provides an in-depth, end-to-end playbook for architects and senior DBAs planning this migration.

Reference Environment

All commands in this article use the following concrete hostnames and roles. Every code block identifies which server and user account the command runs from.

SOURCE: Exadata X7 (Full Rack)
  Compute nodes (X7):
    x7db01.prod.example.com   – DB node 1 (RAC Instance 1)
    x7db02.prod.example.com   – DB node 2 (RAC Instance 2)
  Storage cells (X7):
    x7cel01.prod.example.com, x7cel02.prod.example.com, x7cel03.prod.example.com
    ... (8 cells total for Full Rack)
  Database: ORCL (CDB), running on ASM +DATA_X7 and +RECO_X7
  SCAN:     x7-scan.prod.example.com → 10.10.1.10, .11, .12

TARGET: Exadata X11M (Full Rack)
  Compute nodes (X11M):
    x11db01.prod.example.com  – DB node 1 (new primary RAC Instance 1)
    x11db02.prod.example.com  – DB node 2 (new primary RAC Instance 2)
  Storage cells (X11M):
    x11cel01.prod.example.com ... x11cel08.prod.example.com
  ASM disk groups: +DATA_X11M and +RECO_X11M
  SCAN:     x11-scan.prod.example.com → 10.10.2.10, .11, .12

AUXILIARY HOST (RMAN client / jump host):
  rmanhost.prod.example.com  – separate Linux host with Oracle client installed
  Used for: running RMAN, ZDM, monitoring scripts

Architecture Comparison: X7 vs X11M

Key Implication: InfiniBand → RoCE Migration

The X7 uses InfiniBand for both the compute-to-storage fabric and the RAC private interconnect. The X11M uses RoCE (RDMA over Converged Ethernet). If you are deploying X11M alongside X7 in a temporary co-existence scenario during migration, the two systems cannot share a single cluster interconnect — they require separate networks or dedicated migration connectivity.

Phase 1: Pre-Migration Assessment

1.1 Health Check of the Source X7

Run exachk on the source system a minimum of 4 weeks before migration to identify and resolve issues:

# [[email protected] ~]
# Run full exachk as root (or oracle with sudo)
sudo /opt/oracle.SupportTools/exachk/exachk -a -o all

# Review the HTML report
ls -la /opt/oracle.SupportTools/exachk/exachk_*/

Resolve all FAIL severity findings before migration. Common X7 issues to address:

• Outdated firmware on InfiniBand switches.
• Cell disk I/O errors indicating impending disk failures.
• Inconsistent Grid Infrastructure patch levels across nodes.
• Alert log errors from cell offload processing.

1.2 Workload Characterisation

Capture a comprehensive workload profile of the X7 to establish performance baselines:

# Generate a 2-week AWR report for baseline
sqlplus / as sysdba

SQL> @$ORACLE_HOME/rdbms/admin/awrrpt.sql
-- Choose HTML format, select snap IDs covering peak workload period

-- Capture key performance metrics for baseline documentation
SELECT metric_name,
       ROUND(AVG(average), 2) AS avg_value,
       ROUND(MAX(maxval),  2) AS peak_value,
       metric_unit
FROM dba_hist_sysmetric_summary
WHERE begin_time >= SYSDATE - 14
  AND metric_name IN (
    'DB Time Per Sec',
    'Physical Read Total IO Requests Per Sec',
    'Physical Write Total IO Requests Per Sec',
    'Physical Read Total Bytes Per Sec',
    'Host CPU Utilization (%)',
    'Average Active Sessions',
    'SQL Service Response Time',
    'Cell Physical IO Interconnect Bytes Per Second'
  )
GROUP BY metric_name, metric_unit
ORDER BY metric_name;

Document these baselines. Post-migration, you will compare against them to validate the X11M is performing as expected.

1.3 Inventory of Databases and Configurations

# [[email protected] ~]
olsnodes -n   # list cluster nodes

# Expected output:
# x7db01.prod.example.com  1
# x7db02.prod.example.com  2

srvctl status database -d ORCL
srvctl config database -d ORCL

-- Document database parameter settings (for post-migration validation)
SELECT name, value, description
FROM v$parameter
WHERE isdefault = 'FALSE'
  AND name NOT LIKE 'nls_%'
ORDER BY name;

# Document ASM disk group configuration
asmcmd
ASMCMD> lsdg --discovery
ASMCMD> lsdsk --candidate

Phase 2: Choosing a Migration Approach

There are three main migration approaches. The correct choice depends on tolerable downtime, available infrastructure, and whether both systems are on-site simultaneously.

Approach A: Full Data Copy (Backup/Restore) — Minimal Downtime Risk

Use when: The X7 and X11M are both on-site simultaneously; downtime window of 4–24 hours is acceptable.

1. Provision X11M with Grid Infrastructure and Exadata Software.
2. Create ASM disk groups on X11M.
3. RMAN backup the X7 databases to X11M directly (active duplicate or via backup set to NFS/object storage).
4. During the maintenance window: apply any remaining archive logs (roll forward to current SCN), open the databases on X11M, redirect application connections.

# RMAN active duplicate from X7 to X11M
# (Both systems reachable from RMAN target)
rman auxiliary sys/password@x11m_target

RMAN> DUPLICATE TARGET DATABASE TO ORCL
  FROM ACTIVE DATABASE
  USING BACKUPSET
  SECTION SIZE 5G
  PARALLELISM 16;

Approach B: Oracle Zero Downtime Migration (ZDM) — Sub-minute Downtime

Use when: Downtime must be minimised; GoldenGate licence is available.

As detailed in the OCI migration article, ZDM orchestrates RMAN initial copy + GoldenGate replication. The X11M runs as a target with GoldenGate catching up to the X7. Cutover occurs when GoldenGate lag is near zero.

Approach C: Storage Migration — Cell-level Move

Use when: The databases are very large (multiple TB) and an RMAN copy would take too long.

On modern Exadata, the Rolling Grid Migration feature can migrate ASM disk groups between Exadata machines connected via RDMA, moving data at storage-cell throughput rates (10+ GB/s per cell pair). This is available in Exadata Software 23.x and requires both systems to be accessible.

This is the fastest approach for very large databases and is Oracle’s recommended method for same-generation migrations (e.g., X10M to X11M). For cross-generation migrations from X7 (InfiniBand) to X11M (RoCE), the network difference means standard RMAN or ZDM is typically required.

Phase 3: X11M Deployment and Configuration

3.1 Initial Rack Delivery and Network Configuration

When the X11M arrives:

1. Physical installation: Oracle or a certified partner rack-installs and cables the system.
2. Network verification: The RoCE fabric uses standard Ethernet switches. Verify MTU 9000 (jumbo frames) is configured on all RoCE ports.
3. ILOM / management network: Configure ILOM IP addresses for all compute nodes and storage cells.

3.2 Exadata Software Installation

The X11M ships with Exadata Software 24.x pre-installed. Verify:

# On a cell node: verify Exadata Software version
cellcli -e "LIST CELL DETAIL" | grep softwareVersion

# On a compute node: verify Grid Infrastructure version
$ORACLE_HOME/bin/crsctl query crs activeversion

Apply any mandatory patches identified in the Exadata Quarterly Full Stack Download Patch (QFSDP) before migrating production data.

3.3 Configure ASM Disk Groups on X11M

The X11M delivers more flash capacity than X7. Redesign disk groups to use the available capacity efficiently:

# On X11M: list available cell disks
asmcmd
ASMCMD> lsdsk --candidate --discovery

# Create DATA disk group (NORMAL or HIGH redundancy based on data criticality)
# HIGH redundancy is recommended for production on Exadata (3-way mirroring)

-- Create ASM disk groups from SQL*Plus on the X11M Grid Infrastructure
CREATE DISKGROUP DATA
  HIGH REDUNDANCY
  FAILGROUP fg_cell01 DISK 'o/*/DATA_CD_00_cell01', 'o/*/DATA_CD_01_cell01'
  FAILGROUP fg_cell02 DISK 'o/*/DATA_CD_00_cell02', 'o/*/DATA_CD_01_cell02'
  FAILGROUP fg_cell03 DISK 'o/*/DATA_CD_00_cell03', 'o/*/DATA_CD_01_cell03'
  ATTRIBUTE
    'compatible.asm'   = '19.0',
    'compatible.rdbms' = '19.0',
    'cell.smart_scan_capable' = 'TRUE';

CREATE DISKGROUP RECO
  NORMAL REDUNDANCY
  FAILGROUP fg_cell01 DISK 'o/*/RECO_CD_00_cell01'
  FAILGROUP fg_cell02 DISK 'o/*/RECO_CD_00_cell02'
  FAILGROUP fg_cell03 DISK 'o/*/RECO_CD_00_cell03'
  ATTRIBUTE
    'compatible.asm' = '19.0',
    'cell.smart_scan_capable' = 'TRUE';

3.4 Install Oracle Database Homes

Install Oracle Database 19c (or 21c) binaries on the X11M compute nodes. Do not apply the same patch level as the X7 yet — instead install the latest RU (Release Update) available. The migration is a good opportunity to patch to the latest certified level.

# Unzip database home installer
unzip -q /u01/software/LINUX.X64_193000_db_home.zip -d /u01/app/oracle/product/19.0.0/dbhome_1

# Install Oracle Home
/u01/app/oracle/product/19.0.0/dbhome_1/runInstaller \
  -silent \
  -responseFile /u01/app/oracle/product/19.0.0/dbhome_1/install/response/db_install.rsp

Phase 4: Database Migration Execution (RMAN Active Duplicate)

This section details the Approach A procedure — the most common path for X7-to-X11M.

4.1 Pre-Migration Steps (Last 48 Hours Before Window)

-- [[email protected] ~] as sysdba:
-- Validate database integrity on the X7 source
SELECT * FROM v$database_block_corruption;

# [[email protected] ~]
# RMAN validate — detects corruption before migration
rman target sys/[email protected]:1521/ORCL
RMAN> VALIDATE DATABASE SUMMARY;

# Take a final RMAN backup (safety net)
RMAN> BACKUP DATABASE PLUS ARCHIVELOG TAG 'PRE_MIGRATION_FINAL';

4.2 Active Duplicate During Maintenance Window

# [[email protected] ~]
# rmanhost can reach both x7db01 (source) and x11db01 (auxiliary target)
rman target sys/[email protected]:1521/ORCL \
     auxiliary sys/[email protected]:1521/

RMAN> RUN {
  # Allocate parallel channels on target (X7 — reads from here)
  ALLOCATE CHANNEL s1 DEVICE TYPE DISK;
  ALLOCATE CHANNEL s2 DEVICE TYPE DISK;
  ALLOCATE CHANNEL s3 DEVICE TYPE DISK;
  ALLOCATE CHANNEL s4 DEVICE TYPE DISK;

  # Allocate parallel auxiliary channels (X11M — writes here)
  ALLOCATE AUXILIARY CHANNEL a1 DEVICE TYPE DISK;
  ALLOCATE AUXILIARY CHANNEL a2 DEVICE TYPE DISK;
  ALLOCATE AUXILIARY CHANNEL a3 DEVICE TYPE DISK;
  ALLOCATE AUXILIARY CHANNEL a4 DEVICE TYPE DISK;

  DUPLICATE TARGET DATABASE TO ORCL_X11M
    FROM ACTIVE DATABASE
    USING BACKUPSET
    SECTION SIZE 5G
    DB_FILE_NAME_CONVERT ('+DATA_X7', '+DATA_X11M', '+RECO_X7', '+RECO_X11M')
    SPFILE
      SET db_unique_name='ORCL_X11M'
      SET cluster_database='TRUE'
      SET instance_number='1'
      SET thread='1'
      SET undo_tablespace='UNDOTBS1'
      SET log_archive_dest_1='LOCATION=+RECO_X11M'
    NOOPEN;
}

After the initial copy, apply remaining archive logs from the X7 to bring the X11M up to current SCN:

RMAN> RECOVER DATABASE;
RMAN> ALTER DATABASE OPEN RESETLOGS;

4.3 Post-Open Validation

-- Verify database integrity on X11M
SELECT open_mode, database_role, created FROM v$database;

-- Check for invalid objects
SELECT count(*) FROM dba_objects WHERE status = 'INVALID';

-- Recompile invalids
EXEC DBMS_UTILITY.COMPILE_SCHEMA(schema => 'SCHEMA1', compile_all => FALSE);

-- Run utlrp.sql to recompile all invalid objects in the CDB
@?/rdbms/admin/utlrp.sql

Phase 5: X11M-Specific Tuning

5.1 RoCE Interconnect Verification

# [[email protected] ~]
rdma link
# or
ibstatus   # may still be available as compatibility wrapper

# [[email protected] ~]
oifcfg getif
# Expected output:
# bondib0  192.168.10.0  global  cluster_interconnect
# bond0    10.10.2.0     global  public

-- Verify Oracle RAC interconnect configuration
SELECT name, ip_address, is_public FROM v$cluster_interconnects;

5.2 Adjust SGA for Increased Memory

The X11M has significantly more memory than X7. Resize the SGA/PGA to take advantage:

-- Review current settings
SHOW PARAMETER sga_target;
SHOW PARAMETER pga_aggregate_target;

-- Increase to leverage X11M memory capacity
ALTER SYSTEM SET SGA_TARGET    = 200G SCOPE=BOTH;
ALTER SYSTEM SET SGA_MAX_SIZE  = 200G SCOPE=SPFILE;
ALTER SYSTEM SET PGA_AGGREGATE_TARGET = 50G SCOPE=BOTH;

5.3 Reconfigure IORM for X11M Cell Performance

The X11M cells have higher throughput capacity. Revisit IORM plans — previously set limits may now be conservative:

# [[email protected] ~]
# Or across all cells using dcli from a compute node:
# [[email protected] ~]
dcli -g /u01/app/grid/product/19.0.0/grid/conf/cell_group \
  -l celladmin \
  "cellcli -e 'LIST IORMPLAN DETAIL'"

# Update IORM to reflect new throughput capacity (run on each cell or via dcli)
dcli -g /u01/app/grid/product/19.0.0/grid/conf/cell_group \
  -l celladmin \
  "cellcli -e \"ALTER IORMPLAN active=true, dbplans=({name=default, level=1, allocation=100})\""

5.4 Verify Smart Scan and Cell Offload

-- Run a representative full table scan workload and verify Smart Scan metrics
ALTER SESSION SET cell_offload_processing = true;

SELECT COUNT(*), SUM(amount)
FROM large_fact_table
WHERE sale_date >= DATE '2026-01-01';

-- Check Smart Scan statistics
SELECT n.name,
       ROUND(s.value / 1024 / 1024, 2) AS mb
FROM v$sesstat s
JOIN v$statname n ON s.statistic# = n.statistic#
WHERE s.sid = SYS_CONTEXT('USERENV', 'SID')
  AND n.name IN (
    'cell physical IO bytes eligible for predicate offload',
    'cell physical IO bytes saved by storage index',
    'cell physical IO interconnect bytes returned by smart scan'
  );

Phase 6: Post-Migration Performance Validation

6.1 AWR Comparison

After 1 week on X11M, generate a comparable AWR report and compare against the X7 baseline:

-- Compare key metrics: X7 baseline vs X11M post-migration
SELECT metric_name,
       ROUND(AVG(average), 2) AS avg_value,
       metric_unit
FROM dba_hist_sysmetric_summary
WHERE begin_time >= SYSDATE - 7
  AND metric_name IN (
    'DB Time Per Sec',
    'Physical Read Total IO Requests Per Sec',
    'Physical Read Total Bytes Per Sec',
    'Host CPU Utilization (%)',
    'Average Active Sessions'
  )
GROUP BY metric_name, metric_unit
ORDER BY metric_name;

Compare these values against the X7 baseline documented in Phase 1. Expected X11M improvements:

• CPU utilisation: lower (AMD EPYC has more cores and higher IPC).
• Physical read throughput: higher (PCIe Gen5 NVMe has ~2x raw throughput vs X7 Gen3).
• Smart Scan effectiveness: maintained or improved.

6.2 Run exachk on X11M

# [[email protected] ~]
sudo /opt/oracle.SupportTools/exachk/exachk -a -o all

Address any new findings. The X11M exachk includes X11M-specific checks for RoCE configuration, PCIe Gen5 firmware levels, and AMD EPYC BIOS settings.

6.3 Decommission the X7

Only after:

• 2–4 weeks of stable production operation on X11M.
• All AWR metrics validated against baseline.
• Data Guard (or a physical standby on X11M) confirmed synchronised and tested.
• Application team sign-off on response time metrics.

Then power down and return the X7 hardware per Oracle’s equipment return process.

Migration Timeline Summary