The previous investigation had an abend and an error code to start from. This one is harder precisely because there isn’t one — Replicat is running, lag is zero, the report file is clean, and yet a nightly reconciliation job is flagging rows where the target’s EMAIL column contains what looks like a phone number. When replication looks completely healthy but the data is wrong, the investigation has to start from the data itself, not from a log.
1. The Symptom: Healthy Pipeline, Wrong Data
GGSCI> INFO REPLICAT REP2
REPLICAT REP2 Last Started 2026-07-09 06:02 Status RUNNING
Checkpoint Lag 00:00:02
Log Read Checkpoint File ./dirdat/rt000389
2026-07-09 11:58:41.002011 RBA 41229880
No errors. No discards. Row counts between source and target match exactly for CUSTOMER_PROFILE. This is a silent misapplication — the kind that only shows up in a downstream data-quality check, which is exactly what happened:
Reconciliation Report — CUSTOMER_PROFILE — 2026-07-09
Rows compared: 842,006
Value mismatches: 3,114 rows
Pattern: target.EMAIL contains numeric-only strings resembling target.PHONE values
First mismatch observed: source txn timestamp 2026-07-08 14:22:07
Zone to check first: the first mismatch timestamp, not the current time. This tells you the corruption has a start boundary somewhere in the trail history — something changed at 14:22 on July 8th, and everything before that point is fine.
2. Narrowing Down the Boundary in the Trail
The report file doesn’t go back that far in detail, but checkpoint history does. Look for a checkpoint entry closest to the suspect timestamp to get an approximate RBA to start searching from:
GGSCI> VIEW REPORT REP2
2026-07-08 14:20:03 INFO OGG-01021 Checkpoint file rba 41180212, thread 1
2026-07-08 14:25:11 INFO OGG-01021 Checkpoint file rba 41198740, thread 1
Zone to check: the checkpoint RBA immediately before 14:22:07 — 41180212. That’s the starting position for the Logdump search, not the exact failure point, which we still need to find.
3. Pulling a “Known Good” Record and a “Known Bad” Record
Open the trail and compare a record from well before the boundary against one from after it, side by side.
Logdump 1 > OPEN ./dirdat/rt000388
Logdump 2 > GHDR ON
Logdump 3 > DETAIL DATA
Logdump 4 > USETOKEN ON
Logdump 5 > FILTER TABLE ORDERS.CUSTOMER_PROFILE
Logdump 6 > POS 41150004
Logdump 7 > NEXT
Known-good record (before the boundary):
Table ORDERS.CUSTOMER_PROFILE
Column 1 (x0001), Len 6 CUST_ID : 402981
Column 2 (x0002), Len 12 FIRST_NAME : Isabelle
Column 3 (x0003), Len 9 LAST_NAME : Marchand
Column 4 (x0004), Len 24 EMAIL : [email protected] <-- correct: email-shaped string
Column 5 (x0005), Len 14 PHONE : +33612345678 <-- correct: phone-shaped string
Column 6 (x0006), Len 1 STATUS : A
TDR Index: 41008 (x0000a030)
Now advance past the boundary RBA and pull a record for the same table:
Logdump 8 > POS 41220118
Logdump 9 > NEXT
Suspect record (after the boundary):
Table ORDERS.CUSTOMER_PROFILE
Column 1 (x0001), Len 6 CUST_ID : 507712
Column 2 (x0002), Len 9 FIRST_NAME : Mathieu
Column 3 (x0003), Len 7 LAST_NAME : Girard
Column 4 (x0004), Len 4 EMAIL : GOLD <-- WRONG SHAPE: not an email at all
Column 5 (x0005), Len 22 PHONE : [email protected] <-- WRONG SHAPE: this is an email
Column 6 (x0006), Len 14 STATUS : +33698765432 <-- WRONG SHAPE: this is a phone number
TDR Index: 41008 (x0000a030)
This is the zone that matters most in the whole investigation: the field labels (EMAIL, PHONE, STATUS) haven’t changed, and the TDR Index is identical in both records — but the values have shifted one column to the right of where they belong. GOLD is a loyalty tier, not an email. The data isn’t corrupted; it’s correctly captured, just labeled with the wrong column name.
4. Why the TDR Index Didn’t Change (and Why That’s the Trap)
A shifted TDR Index would have been the easy version of this bug — Logdump would show two different table definitions and the mismatch would be obvious. Here, both records reference the same TDR index, which means Replicat believes the column layout hasn’t changed at all. That’s only possible if Replicat isn’t resolving the table definition dynamically from the trail — it’s reading it from somewhere static.
GGSCI> INFO REPLICAT REP2, DETAIL
...
SOURCEDEFS ./dirdef/customer_profile.def
Zone to check: the presence of a SOURCEDEFS entry at all. A static definitions file, generated once by DEFGEN, is exactly the kind of thing that silently goes stale — it isn’t re-read from the source catalog on every restart, and nothing forces it to be regenerated when the source schema changes.
$ ls -la dirdef/customer_profile.def
-rw-r----- 1 oracle oinstall 2216 Jun 14 09:02 dirdef/customer_profile.def
Zone to check: the file’s modification date, Jun 14, against the app team’s change log:
2026-07-08 13:55 CHANGE-4471 Rebuild CUSTOMER_PROFILE via CTAS to add
LOYALTY_TIER column between LAST_NAME and EMAIL.
Old table renamed and dropped after cutover.
That’s the root cause. On July 8th, the application team rebuilt the table with CREATE TABLE ... AS SELECT to insert a new LOYALTY_TIER column in the middle of the column list (a physical reorg, not a simple ADD COLUMN at the end), then renamed it into place. Extract correctly captured the new physical layout from the redo stream. But Replicat was still interpreting every record against the .def file generated on June 14th — three weeks before the column order changed — so it kept assigning the 4th physical value to EMAIL, the 5th to PHONE, and so on, exactly as the old, now-wrong layout described.
5. The Fix: Regenerate Definitions, Then Reconsider Static Defs Entirely
Regenerate the stale file so it matches current reality:
$GG_HOME/defgen paramfile dirprm/defgen.prm
-- dirprm/defgen.prm
DEFSFILE ./dirdef/customer_profile.def, PURGE
USERIDALIAS source_db
TABLE orders.customer_profile;
Restart Replicat so it picks up the corrected definitions:
GGSCI> STOP REPLICAT REP2
GGSCI> START REPLICAT REP2
New records apply correctly from this point forward. But — and this is the harder part of this investigation compared to a simple abend — the 3,114 already-misapplied rows are still sitting in the target with wrong values. Fixing the definitions file stops new corruption; it does nothing to the rows already written between the 14:22 boundary and the restart. Those need a targeted backfill, re-pulling the affected rows from the source and correcting the target directly — GoldenGate has no mechanism to retroactively reprocess records it already applied “successfully” from its own point of view, since no error was ever raised.
The longer-term fix is avoiding the static file entirely: on 19c+ sources and targets, prefer dynamic/online resolution over SOURCEDEFS where your topology allows it, so table definitions are always read from the live dictionary instead of a point-in-time snapshot that has to be manually kept in sync with every DDL change, including ones — like a physical column reorder — that are easy to forget matter at all.
Quick Reference: What Made This One Harder to Find
| Signal | Abend scenario (previous post) | This scenario |
|---|---|---|
| Error in report file | Yes — pointed straight at the record | None |
| Discard file | Populated with the bad record | Empty |
| Lag | Spiked at the failure | Stayed at zero |
| Starting point for investigation | The error message | A downstream data reconciliation report |
| TDR Index | Unaffected | Identical in good and bad records — the misleading signal |
| Root cause visibility | Directly in the record’s byte length | Only visible by comparing two records across the suspected boundary and checking file timestamps against a change log |
Summary
The dangerous GoldenGate failures aren’t the ones that abend — those stop and wait for you. This one kept running, kept its lag at zero, and quietly mislabeled three thousand rows for hours before anything downstream noticed. The investigative technique that cracked it wasn’t a single Logdump command; it was pulling a record from before a suspected change and a record from after it, comparing them field by field, and noticing that the values had shifted while the metadata claimed nothing had. Whenever data looks wrong but the pipeline looks healthy, that comparison — good record vs. bad record, side by side — is the place to start.