Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
213 changes: 213 additions & 0 deletions docs/innodb-btree-check.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,213 @@
# InnoDB B-tree integrity validation during prepare

## Overview

Percona XtraBackup 8.4.0-6 introduces the [`--check-tables`](xtrabackup-option-reference.md#check-tables) option that validates the structural integrity of InnoDB B-tree indexes during the [`--prepare`](xtrabackup-option-reference.md#prepare) phase. Validation during `--prepare` helps detect corrupted indexes before backup restore or production deployment.

## Why checksum validation is not enough

Percona XtraBackup verifies page checksums during `--backup`. Checksum validation detects physical page corruption, including:

* Torn pages

* Storage bit rot

* Corrupted transfers

* Filesystem-level damage
Comment on lines +11 to +17
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All of them typically mean only one thing. backup is corrupted on disk, after backup is completed. We can simplify


Checksum validation confirms page integrity at the byte level. B-tree structure validation requires additional checks across related pages.
Comment on lines +7 to +19
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Storage bit-rot is an AI word 😄

This is a bit of introduction on what xtrabackup does so far without the --check-tables feature

  1. currently backup ensure every page it copies is valid by verifying the checksum (verify : checksum value of data part should be same the number stored in the page header). InnoDB uses crc32 checksums.

  2. If server and xtrabackup operate on same page, xtrabackup recopies the page until checksum matches. this way backup never copies a invalid checksum page from server. [There is a certain number of retries, after which it will fail the backup. This detail is not very important for now].

  3. A successful backup always ensure checksum-correct backups.

Now we introduce why "checksum-correct backups" are not sufficient.

  1. Disk corruptions/ File system corruptions
    although the data is checksum correct when the backup is taken, a corruption can happen later after the abckup is created. Prepare can detect such corruption only if the page is touched by redo log that xtrabackup copied. This is why it is a good practice to run CHECK TABLE after restore. this ensure all pages are checksum-verified and also the structural integrity is checked.

  2. Pages can be checksum correct but structurally wrong.
    This you mentioned below


Structural corruption that can pass checksum validation includes:

* Broken sibling page links

* Incorrect `PAGE_INDEX_ID` assignments

* Missing or misplaced minimum-record flags

* Invalid parent-to-child page references

* Shared external LOB (large object) pages

* All-zero pages with valid checksums
Comment on lines +21 to +33
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, this section explains what a structural integrity means. I guess this should go before "How --check-tables option work" section


Applying the redo log during `--prepare` copies the existing structural corruption from the source server into the prepared backup. As a result, backups can remain physically consistent while containing logically corrupted indexes.
Copy link
Copy Markdown
Collaborator

@satya-bodapati satya-bodapati May 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, this can happen during the backup. We dont really verify structural integrity during backup. So the corruption can happen during the abckup time.

Can happen during the prepare phase too (if there was such a redo log entry that breaks the structure of index). this is very rare and indicates a server bug. May be we can remove mentioning this possiblity of such corruption occured by applying redo log (prepare phase). A rare thing and complex for user to understand.


## How `--check-tables` works
Copy link
Copy Markdown
Collaborator

@satya-bodapati satya-bodapati May 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, sorry, you wrote it here. I mentioned that How check-tables work section in another comment. Please see if we can integrate something from that comment here.

Either way, we have to decide the order of sections/sub sections

Pre work:
1.How xtrabckup ensures checksum-correct backups (happens during --backup)
2. what do page-checksum correct backup gurantees
3. what page-checksum backups do NOT guarantee
a. the strucutural corruption etc. What does strucutural correctness etc here

Current work

  1. What this feature is
  2. How to use this feature
    a. Use when prepare
    b. Cannot be used during --backup
  3. How this feature works


The `--check-tables` option executes `btr_validate_index()` on every committed index in each `.ibd` tablespace using the number of threads specified by [`--parallel`](xtrabackup-option-reference.md#parallel). `--check-tables` detects structural inconsistencies that page checksum verification cannot detect. This option applies only to InnoDB tables.

Validation runs during the `--prepare` phase after applying the redo log. The process operates in read-only mode against backup files and does not modify backup contents. Validation continues even after detecting corrupted tables, allowing all problematic tables and indexes to be reported in a single run.

The option supports:

* [Parallel execution](#parallel-execution) through [`--parallel`](xtrabackup-option-reference.md#parallel)

* Workflows that use [`--apply-log-only`](xtrabackup-option-reference.md#apply-log-only)

* Transportable tablespace export with [`--export`](xtrabackup-option-reference.md#export).

For each tablespace, Percona XtraBackup:

1. Loads index metadata

2. Identifies committed indexes

3. Executes `btr_validate_index()` on each index

4. Traverses B-tree pages and validates structural relationships

5. Reports detected inconsistencies

The validation process verifies:

* Sibling page relationships

* Parent-to-child page references

* Page ownership metadata

* Minimum-record markers

* External LOB (large object) page ownership

### Offloading `CHECK TABLE`

This option is functionally equivalent to running `CHECK TABLE` on InnoDB tables, but it executes on the backup during the `--prepare` phase instead of on a running production server.

This allows a significant portion of `CHECK TABLE` workload to be offloaded from production systems to an offline environment where the backup is prepared and validated.

### Detected corruption conditions

| Check | Detected condition |
|------|---------------------|
| Broken sibling links | Invalid sibling or parent navigation pointers |
| `PAGE_INDEX_ID` mismatches | Page index ID does not match index metadata |
| Minimum-record flag validation | Minimum-record flag is missing or invalid |
| Parent-child pointer validation | Child page boundaries do not match parent node structure |
| External LOB validation | Shared, freed, or out-of-bounds LOB page references |
| All-zero page detection | Page contains only zero bytes |

### Parallel execution

The `--check-tables` option uses the existing `--parallel` infrastructure in Percona XtraBackup. Worker threads process tablespaces independently.

Each worker thread:

1. Retrieves a tablespace from the shared queue

2. Loads metadata for the tablespace

3. Validates committed indexes

4. Reports validation results

### Limitations

The `--check-tables` option has the following limitations:

* Validation runs only during `--prepare`

* Validation increases CPU and I/O usage on the backup host

* Runtime depends on the number of tablespaces and indexes
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should recommend to use --check-tables on the final prepare only because verify the corruption after every incremental could be slow. --check-tables will verify all the tables every time the option is used.


* Validation does not replace logical consistency checks such as `CHECK TABLE`
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a table compares physical (--check-tables) and logical (CHECK TABLE) checks, highlighting what they detect and what they miss.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

our --check-tables and server's CHECK TABLE are same 😄 Essentially we call the same innodb fucntion on both (btr_validate_index).

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This indeed replaces CHECK TABLE reuqirement for InnoDB Tables only. User don't need to repeat it.

We should also say, this feature doesn't work for MyISAM/RocksDb tables as of now.


## Usage

### Validate a full backup

```bash
xtrabackup --prepare --check-tables \
--target-dir=/backups/full \
--parallel=8
```

### Validate an incremental backup chain

```bash
xtrabackup --prepare --apply-log-only --check-tables \
--target-dir=/backups/full \
--incremental-dir=/backups/inc1 \
--parallel=8
```

### Validate and export tablespaces

```bash
xtrabackup --prepare --export --check-tables \
--target-dir=/backups/full \
--parallel=8
```

## Output

A successful validation operation ends with:

```text
All table checks passed
Copy link
Copy Markdown
Collaborator

@satya-bodapati satya-bodapati May 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sample log:

2026-05-15T15:41:57.808327+01:00 2 [Note] [MY-011825] [Xtrabackup] Checking: mysql/replication_group_member_actions
2026-05-15T15:41:57.808630+01:00 2 [Note] [MY-011825] [Xtrabackup] Checking: mysql/replication_group_configuration_version
2026-05-15T15:41:57.808810+01:00 2 [Note] [MY-011825] [Xtrabackup] Checking: mysql/server_cost
2026-05-15T15:41:57.808998+01:00 2 [Note] [MY-011825] [Xtrabackup] Checking: mysql/engine_cost
2026-05-15T15:41:57.809190+01:00 2 [Note] [MY-011825] [Xtrabackup] Checking: mysql/proxies_priv
2026-05-15T15:41:57.809511+01:00 2 [Note] [MY-011825] [Xtrabackup] Checking: mysql/ndb_binlog_index
2026-05-15T15:41:58.051499+01:00 0 [Note] [MY-011825] [Xtrabackup] All table checks passed.

```

A failed validation operation returns a non-zero exit code and logs the following message:

```text
Copy link
Copy Markdown
Collaborator

@satya-bodapati satya-bodapati May 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sample log when xtrabackup is processing tables.

2026-05-15T13:42:23.691691+01:00 2 [Note] [MY-011825] [Xtrabackup] Checking: test/t1
2026-05-15T13:42:23.697349+01:00 2 [Note] [MY-011825] [Xtrabackup] Checking: test/t_lob
2026-05-15T13:42:23.782555+01:00 2 [Note] [MY-011825] [Xtrabackup] Checking: mysql/dd_properties
2026-05-15T13:42:23.782835+01:00 2 [Note] [MY-011825] [Xtrabackup] Checking: mysql/innodb_dynamic_metadata
2026-05-15T13:42:23.782992+01:00 2 [Note] [MY-011825] [Xtrabackup] Checking: mysql/innodb_table_stats
2026-05-15T13:42:23.783112+01:00 2 [Note] [MY-011825] [Xtrabackup] Checking: mysql/innodb_index_stats
2026-05-15T13:42:23.783276+01:00 2 [Note] [MY-011825] [Xtrabackup] Checking: mysql/innodb_ddl_log
2026-05-15T13:42:23.783446+01:00 2 [Note] [MY-011825] [Xtrabackup] Checking: mysql/catalogs
2026-05-15T13:42:23.783767+01:00 2 [Note] [MY-011825] [Xtrabackup] Checking: mysql/character_sets
2026-05-15T13:42:23.784568+01:00 2 [Note] [MY-011825] [Xtrabackup] Checking: mysql/check_constraints
2026-05-15T13:42:23.785049+01:00 2 [Note] [MY-011825] [Xtrabackup] Checking: mysql/collations
2026-05-15T13:42:23.788787+01:00 2 [Note] [MY-011825] [Xtrabackup] Checking: mysql/column_statistics
2026-05-15T13:42:23.789259+01:00 2 [Note] [MY-011825] [Xtrabackup] Checking: mysql/column_type_elements
2026-05-15T13:42:23.793267+01:00 2 [Note] [MY-011825] [Xtrabackup] Checking: mysql/columns
2026-05-15T13:42:23.927658+01:00 2 [Note] [MY-011825] [Xtrabackup] Checking: mysql/events
2026-05-15T13:42:23.928928+01:00 2 [Note] [MY-011825] [Xtrabackup] Checking: mysql/foreign_key_column_usage
2026-05-15T13:42:23.930218+01:00 2 [Note] [MY-011825] [Xtrabackup] Checking: mysql/foreign_keys
2026-05-15T13:42:23.932221+01:00 2 [Note] [MY-011825] [Xtrabackup] Checking: mysql/index_column_usage

Table check failed. The backup may be corrupted.
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Invidual table names are reported. At the end we have

2026-05-15T13:42:24.670469+01:00 0 [ERROR] [MY-011825] [Xtrabackup] Table check failed. The backup may be corrupted.

```

The log contains detailed information for each detected inconsistency.

### Corruption examples

1. Sibling page relationships corruption.

??? example "Expected output"

```{.text .no-copy}
2026-05-15T13:42:20.270268+01:00 2 [ERROR] [MY-013051] [InnoDB]
In pages [page id: space=2, page number=5]
and [page id: space=2, page number=6]
of index PRIMARY of table test.t1

InnoDB: broken FIL_PAGE_NEXT or FIL_PAGE_PREV links
```

2. Parent-to-child page references corruption.

??? example "Expected output"

```{.text .no-copy}
2026-05-15T13:38:12.343921+01:00 2 [ERROR] [MY-011825] [InnoDB]
B-tree corruption: page 0 is empty but is not the root page
in index PRIMARY. Possible all-zero (unflushed) page.
```

3. Page ownership metadata corruption.

??? example "Expected output"

```{.text .no-copy}
2026-05-15T13:38:12.343894+01:00 2 [ERROR] [MY-011866] [InnoDB]
Page index id 0 != data dictionary index id 204
```

4. Minimum-record markers corruption.

??? example "Expected output"

```{.text .no-copy}
2026-05-15T13:42:27.237530+01:00 2 [ERROR] [MY-014011] [InnoDB]
Minimum record flag is wrongly set to rec on page '4'
at level '0' for index 'PRIMARY' of table 'sys/sys_config'.
```

5. External LOB page ownership corruption.

??? example "Expected output"

```{.text .no-copy}
2026-05-15T13:42:34.475996+01:00 2 [ERROR] [MY-011825] [InnoDB] Invalid record! External LOB first page cannot be shared between two records
2026-05-15T13:42:34.476009+01:00 2 [ERROR] [MY-011825] [InnoDB] The external LOB first page is [page id: space=4294967294, page number=1002]
2026-05-15T13:42:34.476014+01:00 2 [ERROR] [MY-011825] [InnoDB] The first occurrence of the external LOB first page is in record : page_no: 992 with heap_no: 4
```
16 changes: 16 additions & 0 deletions docs/xtrabackup-option-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,22 @@ If a privilege is not needed for the current operation but is missing and may be
xtrabackup: Warning: missing required privilege REPLICATION CLIENT on *.*
```

### check-tables

Usage: `--check-tables`

Introduced in Percona XtraBackup 8.4.0-6.

Enables InnoDB B-tree structural validation during the `--prepare` phase after redo log application completes.

The option runs `btr_validate_index()` on every committed index in each `.ibd` tablespace in the backup. Validation detects structural inconsistencies that page checksum verification does not detect, such as broken B-tree relationships, invalid page metadata, and inconsistent external LOB references.

The validation process runs in read-only mode and does not modify backup data. The operation integrates with `--apply-log-only`, `--parallel`, and `--export`.

If validation detects corruption, xtrabackup reports the issue and returns a non-zero exit code after completing all checks.

Find more information in the [InnoDB B-tree integrity validation during prepare](innodb-btree-check.md) document.

### close-files

Usage: `--close-files`
Expand Down
1 change: 1 addition & 0 deletions mkdocs-base.yml
Original file line number Diff line number Diff line change
Expand Up @@ -206,6 +206,7 @@ nav:


- flush-tables-with-read-lock.md
- innodb-btree-check.md
- log-enhancements.md
- redo-log-archiving.md
- reduction-in-locks.md
Expand Down
Loading