chunked builtin backup engine

[rvrangel]

Description

This is the first PR as part of #20159

This PR adds chunked parallel backup/restore to the builtin backup engine. Files larger than a configurable threshold are split into independently-compressed chunks during backup, which can then be restored in parallel using writes at known offsets.

Changes:

• Two new flags: --builtinbackup-file-chunk-threshold (default 0, disabled) and --builtinbackup-file-chunk-size (default 1GiB)
• During backup, files exceeding the threshold are split into chunks, each stored as a separate object in backup storage
• During restore, chunks of the same file are written concurrently via offsetWriter (pwrite semantics)
• Failed chunks are retried using the same mechanism as whole-file retries
• Backward compatible: threshold=0 disables chunking, and old manifests (no Chunks field) restore identically to before

Related Issue(s)

• Feature Request: Improved builtin backup engine #20159

Checklist

• "Backport to:" labels have been added if this change should be back-ported to release branches
• If this change is to be back-ported to previous releases, a justification is included in the PR description
• Tests were added or are not required
• Did the new or modified tests pass consistently locally and on CI?
• Documentation was added or is not required

Deployment Notes

AI Disclosure

PR created by me with support of Claude, fully tested by me before publishing on our own branch and tested with unit tests and e2e on the main branch

[vitess-bot]

Review Checklist

Hello reviewers! 👋 Please follow this checklist when reviewing this Pull Request.

General

• Ensure that the Pull Request has a descriptive title.
• Ensure there is a link to an issue (except for internal cleanup and flaky test fixes), new features should have an RFC that documents use cases and test cases.

Tests

• Bug fixes should have at least one unit or end-to-end test, enhancement and new features should have a sufficient number of tests.

Documentation

• Apply the release notes (needs details) label if users need to know about this change.
• New features should be documented.
• There should be some code comments as to why things are implemented the way they are.
• There should be a comment at the top of each new or modified test to explain what the test does.

New flags

• Is this flag really necessary?
• Flag names must be clear and intuitive, use dashes (-), and have a clear help text.

If a workflow is added or modified:

• Each item in Jobs should be named in order to mark it as required.
• If the workflow needs to be marked as required, the maintainer team must be notified.

Backward compatibility

• Protobuf changes should be wire-compatible.
• Changes to _vt tables and RPCs need to be backward compatible.
• RPC changes should be compatible with vitess-operator
• If a flag is removed, then it should also be removed from vitess-operator and arewefastyet, if used there.
• vtctl command output order should be stable and awk-able.

[Copilot]

Pull request overview

This PR adds optional chunking to the builtinbackupengine so that large MySQL files can be backed up and restored as independently-compressed pieces, enabling higher parallelism (especially beneficial for object stores like S3) and improving restore throughput.

Changes:

• Introduces chunk metadata in the backup manifest (FileEntry.Chunks) and new flags to control chunking (--builtinbackup-file-chunk-threshold, --builtinbackup-file-chunk-size).
• Updates builtin backup/restore to split large files into chunks for parallel backup and to restore chunked files via parallel WriteAt (pwrite-style) writes into a pre-sized destination.
• Adds unit and end-to-end tests validating chunk name parsing and verifying chunked vs non-chunked backups via MANIFEST inspection.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
go/vt/mysqlctl/file_close_test.go	Updates tests for the new backupFile(..., chunkIndex) signature.
go/vt/mysqlctl/builtinbackupengine.go	Core implementation: chunking flags, manifest schema, chunked backup work scheduling, and parallel chunk restore.
go/vt/mysqlctl/builtinbackupengine_test.go	Adds unit tests for parsing storage names (parseBackupName).
go/test/endtoend/backup/vtctlbackup/backup_utils.go	Adds helpers to verify chunking by reading MANIFEST and counting chunks.
go/test/endtoend/backup/vtctlbackup/backup_test.go	Adds end-to-end tests for chunked and non-chunked builtin backups with forced small thresholds/sizes.
go/flags/endtoend/vttestserver.txt	Documents new builtinbackup chunking flags in end-to-end flag snapshots.
go/flags/endtoend/vttablet.txt	Documents new builtinbackup chunking flags in end-to-end flag snapshots.
go/flags/endtoend/vtctld.txt	Documents new builtinbackup chunking flags in end-to-end flag snapshots.
go/flags/endtoend/vtcombo.txt	Documents new builtinbackup chunking flags in end-to-end flag snapshots.
go/flags/endtoend/vtbackup.txt	Documents new builtinbackup chunking flags in end-to-end flag snapshots.

Comments suppressed due to low confidence (2)

go/vt/mysqlctl/builtinbackupengine.go:1372

• Chunk restore goroutines close over loop variables j and fe (and use dest/fe.Name inside the closure). This can result in writing the wrong chunk offset/data and misreporting errors/logs. Rebind the loop variables (e.g. j := j, feLocal := fe) before starting each goroutine.

			for j := range fe.Chunks {
				g.Go(func() error {
					chunk := &fe.Chunks[j]

					select {

go/vt/mysqlctl/builtinbackupengine.go:1394

• Non-chunked restore goroutine closes over i/fe from the enclosing for-loop. This can cause it to restore the wrong file index and log/record errors under the wrong name. Capture locals (e.g. iLocal := i, feLocal := fe) before g.Go.

			// Non-chunked file: restore as before.
			g.Go(func() error {
				name := strconv.Itoa(i)

				select {

[promptless]

Promptless prepared a documentation update related to this change.

Triggered by PR #20167

Added documentation for the new --builtinbackup-file-chunk-threshold and --builtinbackup-file-chunk-size flags to the backup and restore overview guide. These flags enable parallel backup and restore of large files by splitting them into independently-compressed chunks.

Review: Document builtin backup chunking flags

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

[codecov]

Codecov Report

❌ Patch coverage is 80.58608% with 53 lines in your changes missing coverage. Please review.
✅ Project coverage is 52.95%. Comparing base (70c7a72) to head (29ad57c).
⚠️ Report is 352 commits behind head on main.

Files with missing lines	Patch %	Lines
go/vt/mysqlctl/builtinbackupengine.go	80.58%	53 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (70c7a72) and HEAD (29ad57c). Click for more details.

HEAD has 1 upload less than BASE

Flag	BASE (70c7a72)	HEAD (29ad57c)
	1	0

Additional details and impacted files

@@             Coverage Diff             @@
##             main   #20167       +/-   ##
===========================================
- Coverage   69.67%   52.95%   -16.72%     
===========================================
  Files        1614       46     -1568     
  Lines      216793     7315   -209478     
===========================================
- Hits       151044     3874   -147170     
+ Misses      65749     3441    -62308     

Flag	Coverage Δ
partial	52.95% <80.58%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 5 comments.

[mattlord]

go/vt/mysqlctl/builtinbackupengine.go:1386-1390 ignores final close errors for chunked restore destinations. Non-chunked restore propagates destination close failures because they can mean data was not safely flushed; chunked restore only logs them and can report success, then later attempt to start MySQL on an incomplete/corrupt file. Please collect these close errors and return them, and also check the dest.Close() in createChunkedDestinations at line 1367.

go/vt/mysqlctl/builtinbackupengine.go:196-198 / :691-694 has no bound on the chunk count. A small --builtinbackup-file-chunk-size typo, e.g. 1, can allocate one FileChunk and one work item per byte of a large InnoDB file before backup starts. Please enforce a sane minimum chunk size or a max chunks-per-file limit before allocating.

go/vt/mysqlctl/builtinbackupengine.go:281-283 validates chunk size even when chunking is disabled or when taking an incremental backup that may not use chunking. I’d validate chunk-size > 0 only when chunk-threshold > 0, reject negative thresholds explicitly, and fix the message to say must be > 0.

I agree with the compatibility caveat too: once chunking is enabled, those backups are not restorable by older Vitess versions because old restore code ignores Chunks and looks for whole-file objects. That should be called out in release notes summary.

[rvrangel]

@mattlord PR updated!

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 4 comments.

[mattlord]

Just one (yay!) note... In go/vt/mysqlctl/builtinbackupengine.go:1329-1359 we should preserve chunked-destination close failures across retries. restoreFileEntries can return both retryable per-chunk errors from bh.Error() and direct errors from closing the shared chunked destination file (:1419-1424, after return bh.Error() at :1507-1508). restoreFiles then retries only bh.GetFailedFiles() and returns success if that retry succeeds, which drops the first-pass close failure. A close error can be where delayed writeback/ENOSPC is reported, so retrying only one failed chunk does not prove the already-written chunks are safe. Please keep direct restore errors separate and return them after retry, or mark the whole affected file/chunks for retry. A regression test should combine one retryable chunk failure with a chunked destination close failure.

Thanks, @rvrangel !

[rvrangel]

@mattlord if we hit a close error (specially for ENOSPC but also for failed flushes to disk) it might be better to skip retries altogether as it is very likely the same issue will fail again.

what do you think if on the deferred function (builtinbackupengine.go#L1419-L1426) we just wrapped the finalErr in a errRestoreFatal and that will prevent the restore from going forward?

[rvrangel]

added the above fix and a test for it too

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 3 comments.

[mattlord]

In 1go/vt/mysqlctl/builtinbackupengine.go:717I think that we should preserve first-pass backupWorkItems errors. The first backup pass still seems to discard thebackupWorkItems’ return value, but the retry path treats that same return as significant. If bh.EndBackup(ctx)returns a direct error that is not represented inbh.GetFailedFiles()`, this can continue into MANIFEST upload and report a successful backup. Please capture the first-pass error and return it when there are no per-file failures to retry, or preserve it across retry if needed. Unless I'm missing something?

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.

[frouioui]

This is forward compatible though right? Taking as an example a vtbackup job running periodically, if vtbackup starts being configured to use chunks will it be able to restore the previous non-chunked backup?

[rvrangel]

yes, that is correct. if the Chunks field in the MANIFEST is empty, it will use the same approach and functions as before.

[rvrangel]

I added a new e2e test that takes a back without chunking and restores it on a replica with chunking enabled in 29ad57c

[frouioui]

Any reason to make this a non-constant?

[rvrangel]

this was done to avoid the exponential back-off on one of the tests by changing the retry value:

https://github.com/rvrangel/vitess/blob/27325e4a9d13cde5add405e3c1d1d227825d25cf/go/vt/mysqlctl/builtinbackupengine_test.go#L494-L509

[frouioui]

We infer the chunk index via j, the position of the chunk in the slice of chunks (fe.Chunks). fes can be built during a retry in the function restoreFiles before being sent down to restoreFileEntries.

The retry-failed-files code path in restoreFiles builds a new fes using only failed files/chunks. It is not impossible that a chunk that was previously in the index e.g. 10 now becomes index e.g. 0. We would run into a similar scenario each time a subset of chunks failed and are being retried.

To keep the chunk index consistent, can we infer its index another way than by looking at the slice index?

[rvrangel]

ah, good point! We actually don't use the index for restoring, just when we are logging, so we can drop it and usage the StorageName of the chunk instead

[frouioui]

I would like to see more E2E tests for this with these characteristics:

• Assertions on a deterministic amount of chunks and files
• Large amount of chunks
• With chunked and unchunked files
• With different storage handlers, like S3 using MinIO (e.g. go/test/endtoend/backup/s3/s3_builtin_test.go)
• Using vtbackup
• With failure scenarios
• Showing the forward compatibility of restoring an unchunked backup with chunks parameters
• Showing the failure scenario of what happens when a chunked backup is restored without the chunks parameters

Can you also provide a performance comparison between chunking and not-chunking. Mostly to get an idea of what the performance impact can be on a stable cluster and hardware.

[rvrangel]

@frouioui ack, let me take a look to see if I can cover most of that this week

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated no new comments.