Skip to content

Commit b164402

Browse files
patrickbirchpatrickbirch
committed
PXB-3144 [DOCS] - update How XtraBackup works 8.4
 modified: docs/how-xtrabackup-works.md
1 parent f7f5f6f commit b164402

8 files changed

Lines changed: 329 additions & 111 deletions

‎docs/how-xtrabackup-works-explanation.md‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
# Explanation: how Percona XtraBackup achieves consistency
2+
3+
PXB’s behavior—redo, undo, locks, and how far “lockless” can go. Procedures: [How-to: backup lifecycle operations](how-xtrabackup-works-how-to.md). Facts and defaults: [Reference: backup lifecycle](how-xtrabackup-works-reference.md).
4+
5+
## Why a live copy isn’t consistent yet
6+
7+
InnoDB keeps writing while PXB reads files, so the copied pages alone are not one committed state. PXB adds a continuous redo capture from backup start to end—physical page changes. `--prepare` then replays that redo on the copy: same idea as InnoDB crash recovery, but offline on the backup tree.
8+
9+
Consistency appears in the prepare step, not in the raw copy.
10+
11+
## The redo thread
12+
13+
PXB stamps an LSN and copies InnoDB data while a background thread streams redo for the whole run. Redo files wrap and recycle; the thread must keep every record prepare will need through the backup’s end point. Data files plus captured redo let prepare roll pages forward to one moment.
14+
15+
Optional `--register-redo-log-consumer` registers PXB so the server won’t purge redo before PXB reads redo—vital on brutally write-heavy hosts—at the cost of more redo on disk and tighter free space.
16+
17+
## Why locking splits into three beats
18+
19+
PXB stages work so InnoDB DML keeps moving during the bulk copy:
20+
21+
1. Open copy: copy InnoDB data and redo while transactions run.
22+
23+
2. Backup lock: copy non-InnoDB engines without a coarse global read lock. DDL tightens here; exactly what is blocked depends on the server.
24+
25+
3. Short binlog lock: `LOCK BINLOG FOR BACKUP` briefly freezes coordinate changes while PXB reads `performance_schema.log_status` and finishes coordination redo.
26+
27+
`--lock-ddl=ON` grabs the backup lock immediately; `--lock-ddl=REDUCED` waits until InnoDB is copied—you trade how long DDL stays restricted against when DDL can race the InnoDB pass.
28+
29+
## When “lockless” is even possible
30+
31+
PXB can skip backup locks only if every table in every schema—including `mysql`—sits on InnoDB. Most sites still have CSV or MyISAM in `mysql` (e.g. `general_log`), so truly lockless runs are rare unless you’ve forced all storage to InnoDB.
32+
33+
Replication quirks differ by build: Percona Server may fold relay coordinates into `log_status`, easing `--slave-info`; stock MySQL can still demand `FLUSH TABLES WITH READ LOCK` for some relay-position needs.
34+
35+
## Prepare: redo alone isn’t enough
36+
37+
Redo replay is physical: logged page changes get reapplied. Those records can include uncommitted work—the server may have flushed dirty pages and redo for transactions still open at backup end.
38+
39+
After redo, undo plus Serialized Dictionary Information (SDI) in the tablespaces drives the logical rollback of that in-flight work (SDI stands in for old `.frm` layout during rollback). When prepare finishes, InnoDB matches the backup’s close-out point with open transactions stripped out; non-InnoDB files already match because backup copied them under the right locks.
40+
41+
## Cloud and streaming
42+
43+
`xbcloud` or streaming doesn’t change the mental model—slow networks just stretch Final Sync, so short locks can linger. Plan longer critical sections.
44+
45+
## See also
46+
47+
* [Tutorial: walk through a physical backup](how-xtrabackup-works-tutorial.md)
48+
49+
* [How Percona XtraBackup works](how-xtrabackup-works.md) (hub)

‎docs/how-xtrabackup-works-how-to.md‎

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
# Backup lifecycle
2+
3+
* Grant only the required privileges for your workflow.
4+
5+
* Run backup, prepare, and restore in the right order.
6+
7+
* Capture binary log coordinates for recovery and replication tasks.
8+
9+
* Choose operational flags such as `--lock-ddl` and `--register-redo-log-consumer` based on workload and lock tolerance.
10+
11+
## Grant the minimum privileges
12+
13+
1. Give the backup user `BACKUP_ADMIN` so PXB can read `performance_schema.log_status` and take backup locks (`LOCK INSTANCE FOR BACKUP`, `LOCK TABLES FOR BACKUP`, or `LOCK BINLOG FOR BACKUP`) as your server and options require.
14+
15+
2. Add `RELOAD`, `LOCK TABLES`, and/or `REPLICATION CLIENT` only when the workflow needs them (for example, `FLUSH TABLES WITH READ LOCK` or `--slave-info`).
16+
17+
See lists and examples in [Connection and privileges needed](privileges.md).
18+
19+
## Run a backup and capture binary log coordinates
20+
21+
1. Run `xtrabackup` with your target directory and usual options.
22+
23+
2. Redirect STDERR to save binary log position to a file (PXB prints coordinates there), for example `xtrabackup … 2> backupout.log`
24+
25+
3. Verify exit code `0` on success.
26+
27+
See the file list created by a backup in [Index of files created by Percona XtraBackup](xtrabackup-files.md).
28+
29+
## Pick `--lock-ddl` (backup locks)
30+
31+
When backup locks exist, PXB uses those locks to copy non-InnoDB files without stalling InnoDB DML.
32+
33+
| Option | Effect |
34+
|--------|--------|
35+
| `--lock-ddl=ON` (default) | Take the backup lock at start. DDL stays blocked for the full run (unless you change the workflow). |
36+
| `--lock-ddl=REDUCED` | Take the lock after InnoDB is copied. This gives a shorter DDL block with a different trade-off vs DDL during the InnoDB phase. |
37+
38+
See a compact comparison in [Reference: backup lifecycle](how-xtrabackup-works-reference.md).
39+
40+
## Turn on `--register-redo-log-consumer` or not
41+
42+
Enable the option when write throughput is high enough that redo might be purged before PXB finishes reading redo—cuts failures from missing redo.
43+
44+
Review these checks before enabling the option.
45+
46+
* Plan for redo bloat and a full disk if the server retains redo longer.
47+
48+
* Monitor free space and I/O during the run.
49+
50+
* Abort the backup (Ctrl+C or SIGTERM to `xtrabackup`) if disk is critical so the consumer releases and the server can purge redo. Free space, then retry.
51+
52+
Default is off—enable only with spare disk.
53+
54+
## Prepare (`--prepare`)
55+
56+
1. Run `xtrabackup --prepare` with `--target-dir` set to the backup directory.
57+
58+
2. On large or incremental backups, tune prepare performance with:
59+
60+
* Use `--use-memory` (often 1G–2G when RAM allows, and the default is small).
61+
62+
* Use `--parallel` for `.delta` files on incremental backups (8.4.0-3+). The `--parallel` option does not parallelize first-pass redo on a full backup.
63+
64+
```bash
65+
xtrabackup --prepare --use-memory=2G --parallel=4 --target-dir=/data/backups/
66+
```
67+
68+
If prepare shares the box with production `mysqld`, leave RAM for the OS and server—OOM kills happen.
69+
70+
[`--use-memory`](xtrabackup-option-reference.md#use-memory), [`--parallel`](xtrabackup-option-reference.md#parallel).
71+
72+
## Restore (`--copy-back` or `--move-back`)
73+
74+
1. Stop `mysqld` if mysqld owns the destination datadir.
75+
76+
2. Run `xtrabackup --copy-back` or `--move-back` with `--target-dir` pointing at the prepared backup. PXB reads paths from config (`datadir`, InnoDB paths, log paths, and related settings).
77+
78+
3. Fix ownership and permissions before starting `mysqld` (for example, `chown -R mysql:mysql /var/lib/mysql`). Backup files belong to whoever ran `xtrabackup`, and the server expects the `mysql` user.
79+
80+
4. Start `mysqld`.
81+
82+
Use `--move-back` only when space is tight—the command deletes files from the backup directory and consumes that backup.
83+
84+
See full restore flows in [Restore full, incremental, and compressed backups](restore-a-backup.md).
85+
86+
## Stream or send backups to the cloud
87+
88+
The steps are the same. Slow networks stretch Final Sync, so binary log locks can last longer—size your window and capacity.
89+
90+
* [Take a streaming backup](take-streaming-backup.md)
91+
92+
* [xbcloud binary overview](xbcloud-binary-overview.md)

‎docs/how-xtrabackup-works-reference.md‎

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
# Backup lifecycle
2+
3+
Quick-scan facts—privileges, phases, defaults, behavior. For a walkthrough see the [tutorial](how-xtrabackup-works-tutorial.md); for procedures see the [how-to](how-xtrabackup-works-how-to.md).
4+
5+
6+
## Privileges (summary)
7+
8+
| Privilege | Typical use |
9+
|-----------|-------------|
10+
| `BACKUP_ADMIN` | Read `performance_schema.log_status`; use `LOCK INSTANCE FOR BACKUP`, `LOCK TABLES FOR BACKUP`, `LOCK BINLOG FOR BACKUP` when applicable |
11+
| `RELOAD` | Workflows using `FLUSH TABLES WITH READ LOCK` or similar |
12+
| `LOCK TABLES` | Workflows that issue table-level locks |
13+
| `REPLICATION CLIENT` | `--slave-info` and related replication metadata |
14+
15+
[Connection and privileges needed](privileges.md) has the full list.
16+
17+
## Three operator-facing phases
18+
19+
| Step | Phase | What happens |
20+
|------|-------|----------------|
21+
| 1 | Backup (hot copy) | Copy data files; stream redo for the whole run |
22+
| 2 | Prepare | Replay redo; roll back uncommitted work |
23+
| 3 | Restore | `--copy-back` or `--move-back` into `datadir` |
24+
25+
## Three in-backup sub-phases (locking)
26+
27+
With backup locks, Percona XtraBackup (PXB) typically runs the stages in the table below. Each numbered row is a phase in order: the Locks column is the global lock posture for that phase, and the Work column is what PXB copies or records while those locks apply—InnoDB pages and redo first (often with no global backup lock), then non-InnoDB files under the backup lock (with DDL restricted), then a short `LOCK BINLOG FOR BACKUP` to pin binlog or replica coordinates. Exact steps depend on server version and which storage engines are in use.
28+
29+
| # | Locks | Work |
30+
|---|--------|------|
31+
| 1 | None global | Copy InnoDB files and redo while data manipulation language (DML) runs |
32+
| 2 | Backup lock | Copy non-InnoDB files (e.g. `.frm`, `.MRG`, `.MYD`, `.MYI`, `.CSM`, `.CSV`, `.sdi` (serialized dictionary information), `.par`); data definition language (DDL) restricted, DML usually allowed—details are server-specific |
33+
| 3 | `LOCK BINLOG FOR BACKUP` (short) | Pin binlog/replica coordinates; read `performance_schema.log_status`; drop locks |
34+
35+
Backup locks vs FTWRL (`FLUSH TABLES WITH READ LOCK`): [Percona Server backup locks](https://docs.percona.com/percona-server/innovation-release/backup-locks.html).
36+
MySQL {{vers}}: [`LOCK INSTANCE FOR BACKUP`](https://dev.mysql.com/doc/refman/{{vers}}/en/lock-instance-for-backup.html).
37+
38+
## `--lock-ddl`
39+
40+
| Value | When the backup lock is taken (typical) |
41+
|-------|----------------------------------------|
42+
| `ON` (default) | Backup start |
43+
| `REDUCED` | After InnoDB data copy finishes |
44+
45+
`ON` blocks DDL for the whole window from the beginning; `REDUCED` delays that until InnoDB is copied (shorter DDL stall, different overlap with DDL during the InnoDB phase). InnoDB DML usually keeps running through the main copy.
46+
47+
## `--register-redo-log-consumer`
48+
49+
| Field | Detail |
50+
|-------|--------|
51+
| Default | Off |
52+
| Does | Registers PXB as a redo consumer so the server won’t purge a redo file until PXB copies that file |
53+
| Cost | Redo can pile up (“redo bloat”); disk use may spike |
54+
| Writes | Server may stall writes briefly while the consumer advances |
55+
56+
## Prepare options (subset)
57+
58+
| Option | Notes |
59+
|--------|--------|
60+
| `--use-memory` | Random-access memory (RAM) for prepare only (default 100MB); larger often helps |
61+
| `--parallel` | 8.4.0-3+: parallel `.delta` apply for incrementals; not the same as parallel full-backup redo replay |
62+
63+
Full flags: [xtrabackup option reference](xtrabackup-option-reference.md).
64+
65+
## Restore (facts)
66+
67+
* Pulls paths from `my.cnf` (e.g. `datadir`, `innodb_data_home_dir`, `innodb_data_file_path`, `innodb_log_group_home_dir`).
68+
69+
* Order: MyISAM-family files first (`.MRG`, `.MYD`, `.MYI`, `.CSM`, `.CSV`, `.sdi`, `.par`), then InnoDB tables/indexes, then logs.
70+
71+
* Keeps file attributes.
72+
73+
* Successful backup prints binlog coordinates to standard error (STDERR)—redirect if you need a file.
74+
75+
## When backup locks are skipped (“lockless”)
76+
77+
Locks stay off only if every table in every schema—including `mysql`—is InnoDB. Commonly `mysql` still holds CSV/MyISAM tables (e.g. `general_log`), so PXB usually takes backup locks anyway.
78+
79+
| Server | Notes |
80+
|--------|--------|
81+
| Percona Server for MySQL {{vers}} | `log_status` may carry relay coordinates; `--slave-info` can skip extra locks |
82+
| Standard MySQL {{vers}} | May still need `FLUSH TABLES WITH READ LOCK` with `--slave-info` when you need relay position |
83+
84+
## Cloud / streaming
85+
86+
Same phases; slow networks stretch Final Sync and can lengthen short locks. [Streaming backup](take-streaming-backup.md), [xbcloud overview](xbcloud-binary-overview.md).
87+
88+
## See also
89+
90+
* [Index of files created by Percona XtraBackup](xtrabackup-files.md)
91+
92+
* [Restore full, incremental, and compressed backups](restore-a-backup.md)

‎docs/how-xtrabackup-works-tutorial.md‎

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
# PXB in practice: backup, prepare, restore
2+
3+
Follow PXB’s order—backup, prepare, restore—and you’ll see where consistency appears. Use the [hub](how-xtrabackup-works.md) to jump to runbooks, tables, or deeper theory.
4+
5+
## Outcomes
6+
7+
You’ll be able to:
8+
9+
* Name the three phases: backup, prepare, restore.
10+
11+
* Say why a live file copy isn’t consistent yet, and what makes the dataset consistent.
12+
13+
* Explain why the redo thread runs in parallel with the file copy, and why `.ibd` files alone would not suffice.
14+
15+
* Describe roll forward versus roll back, and how undo plus SDI handle transactions still open when backup ends.
16+
17+
* Sketch when `--register-redo-log-consumer` helps on busy servers, and why redo bloat threatens disk space.
18+
19+
* Contrast `--lock-ddl=ON` with `--lock-ddl=REDUCED` in terms of when the backup lock appears.
20+
21+
## Step 1 — Hot copy (backup)
22+
23+
PXB copies InnoDB and other files while MySQL keeps serving traffic. The copy takes time, so on-disk pages at any instant don’t match one commit point—that’s normal. You still can’t start `mysqld` on that raw tree.
24+
25+
PXB records a start LSN and runs a redo thread until backup ends. That stream records every change in the window so a later step can bring the copied pages current.
26+
27+
Takeaway: backup holds files plus redo, not a frozen disk image.
28+
29+
### Why a parallel redo stream matters
30+
31+
The `.ibd` read spans real clock time: pages copied early reflect older LSNs than pages copied later. The redo thread exists so PXB also captures every InnoDB change from backup start through backup end. Without that continuous redo capture, there would be no complete log to replay and converge on a single consistent point.
32+
33+
If PXB copied only `.ibd` data files and skipped redo, the directory would hold physically mixed page images with no way to run InnoDB’s normal recovery to one LSN—the dataset would not be a valid starting point for a server.
34+
35+
### When `--register-redo-log-consumer` enters the picture
36+
37+
On a high-traffic host, redo files can rotate or get purged before PXB finishes reading them, which risks a failed or unusable backup. Optional `--register-redo-log-consumer` registers PXB so the server retains redo until PXB has copied the needed files (see [How-to: backup lifecycle operations](how-xtrabackup-works-how-to.md)). The trade-off is redo bloat: redo stays on disk longer while backup runs, so a long backup on a heavy-write system can consume a lot of space—monitor free disk closely.
38+
39+
### `--lock-ddl=ON` versus `--lock-ddl=REDUCED`
40+
41+
Both modes relate to when PXB takes the backup lock used for copying non-InnoDB files without stopping InnoDB DML.
42+
43+
* With `--lock-ddl=ON` (default), the backup lock is taken at backup start, so DDL faces restriction from the first moment of the run.
44+
45+
* With `--lock-ddl=REDUCED`, the backup lock is taken only after InnoDB data has been copied, which shortens how long DDL must wait under that lock, but changes overlap with DDL during the InnoDB phase.
46+
47+
Compact timing table: [Reference: backup lifecycle](how-xtrabackup-works-reference.md).
48+
49+
## Step 2 — Prepare (make the dataset consistent)
50+
51+
After backup completes, the target directory is still raw: InnoDB pages reflect mixed points in time even though redo for the whole window sits beside them. A MySQL server must not use that tree as a live datadir until `--prepare` runs.
52+
53+
`--prepare` performs two coordinated steps:
54+
55+
* Roll forward: replay captured redo onto the copied tablespaces (physical page updates).
56+
57+
* Roll back: remove changes from transactions that had not committed at the backup end point. Redo may have replayed those changes; undo records describe how to reverse row-level effects, and Serialized Dictionary Information (SDI) in the tablespaces supplies table and index definitions so rollback can interpret undo logically (the role older `.frm` files once played).
58+
59+
Non-InnoDB tables match the same moment because PXB copied them under the right locks during backup.
60+
61+
Takeaway: the prepare step turns raw files plus redo into a crash-recovery-consistent dataset—the same class of state InnoDB expects after offline recovery—not the copy step alone.
62+
63+
## Step 3 — Restore (deploy)
64+
65+
`--copy-back` or `--move-back` puts the prepared files in the datadir; set ownership and permissions, then start `mysqld`.
66+
67+
Takeaway: restore mostly moves files; the prepare step did the hard part.
68+
69+
## How the phases fit together
70+
71+
InnoDB crash recovery does the same thing online: redo advances pages, undo drops half-finished transactions. The backup workflow runs that pipeline offline on the backup.
72+
73+
## Where to go next
74+
75+
| Goal | Page |
76+
|------|------|
77+
| Run commands, pick options | [How-to: backup lifecycle operations](how-xtrabackup-works-how-to.md) |
78+
| Look up privileges and phases | [Reference: backup lifecycle](how-xtrabackup-works-reference.md) |
79+
| Redo, undo, locks, lockless | [Explanation: how PXB achieves consistency](how-xtrabackup-works-explanation.md) |

0 commit comments

Comments
 (0)