Skip to content

crl-release-25.3: db: fix BlobBytesCompacted accounting #5447

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Oct 20, 2025
Merged

crl-release-25.3: db: fix BlobBytesCompacted accounting #5447

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 3 additions & 4 deletions compaction.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3448,17 +3448,16 @@ func (c *tableCompaction) makeVersionEdit(result compact.Result) (*manifest.Vers
// TODO(jackson): This BytesRead value does not include any blob files
// written. It either should, or we should add a separate metric.
outputMetrics.TableBytesRead = c.outputLevel.files.TableSizeSum()
outputMetrics.BlobBytesCompacted = result.Stats.CumulativeBlobFileSize
if c.flush.flushables != nil {
outputMetrics.BlobBytesFlushed = result.Stats.CumulativeBlobFileSize
}
if len(c.extraLevels) > 0 {
outputMetrics.TableBytesIn += c.extraLevels[0].files.TableSizeSum()
}
outputMetrics.TableBytesRead += outputMetrics.TableBytesIn

if len(c.flush.flushables) == 0 {
c.metrics.perLevel.level(c.startLevel.level)
outputMetrics.BlobBytesCompacted = result.Stats.CumulativeBlobFileSize
} else {
outputMetrics.BlobBytesFlushed = result.Stats.CumulativeBlobFileSize
}
if len(c.extraLevels) > 0 {
c.metrics.perLevel.level(c.extraLevels[0].level)
Expand Down
8 changes: 4 additions & 4 deletions testdata/compaction/value_separation
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,14 +123,14 @@ metrics
| | | | ingested | moved | written | | amp | val sep | multilevel
level | tables size val-bl vtables | score ff cff | in | tables size | tables size | tables size | read | r w | refsz valblk | top in read
------+-----------------------------+----------------+-------+--------------+--------------+--------------+-------+----------+---------------+------------------
0 | 0 0B 0B 0 | - 0 0 | 41B | 0 0B | 0 0B | 1 861B | 0B | 0 25.9 | 0B 0B | 0B 0B 0B
0 | 0 0B 0B 0 | - 0 0 | 41B | 0 0B | 0 0B | 1 861B | 0B | 0 23.4 | 0B 0B | 0B 0B 0B
1 | 0 0B 0B 0 | - 0 0 | 0B | 0 0B | 1 861B | 0 0B | 0B | 0 0 | 0B 0B | 0B 0B 0B
2 | 0 0B 0B 0 | - 0 0 | 0B | 0 0B | 0 0B | 0 0B | 0B | 0 0 | 0B 0B | 0B 0B 0B
3 | 0 0B 0B 0 | - 0 0 | 0B | 0 0B | 1 861B | 0 0B | 0B | 0 0 | 0B 0B | 0B 0B 0B
4 | 0 0B 0B 0 | - 0 0 | 0B | 0 0B | 0 0B | 0 0B | 0B | 0 0 | 0B 0B | 0B 0B 0B
5 | 0 0B 0B 0 | - 0 0 | 0B | 0 0B | 1 861B | 0 0B | 0B | 0 0 | 0B 0B | 0B 0B 0B
6 | 1 931B 0B 0 | - 0.00 0.00 | 861B | 0 0B | 0 0B | 1 931B | 1.7KB | 1 1.21 | 112B 0B | 0B 0B 0B
total | 1 931B 0B 0 | - - - | 41B | 0 0B | 3 2.5KB | 2 1.8KB | 1.7KB | 1 52.3 | 112B 0B | 0B 0B 0B
total | 1 931B 0B 0 | - - - | 41B | 0 0B | 3 2.5KB | 2 1.8KB | 1.7KB | 1 49.9 | 112B 0B | 0B 0B 0B
----------------------------------------------------------------------------------------------------------------------------------------------------------------
WAL: 1 files (0B) in: 30B written: 41B (37% overhead)
Flushes: 1
Expand Down Expand Up @@ -363,14 +363,14 @@ metrics
| | | | ingested | moved | written | | amp | val sep
level | tables size val-bl vtables | score ff cff | in | tables size | tables size | tables size | read | r w | refsz valblk
------+-----------------------------+----------------+-------+--------------+--------------+--------------+-------+----------+--------------
0 | 0 0B 0B 0 | - 0 0 | 156B | 0 0B | 0 0B | 2 1.8KB | 0B | 0 14.8 | 0B 0B
0 | 0 0B 0B 0 | - 0 0 | 156B | 0 0B | 0 0B | 2 1.8KB | 0B | 0 13.2 | 0B 0B
1 | 0 0B 0B 0 | - 0 0 | 0B | 0 0B | 0 0B | 0 0B | 0B | 0 0 | 0B 0B
2 | 0 0B 0B 0 | - 0 0 | 0B | 0 0B | 0 0B | 0 0B | 0B | 0 0 | 0B 0B
3 | 0 0B 0B 0 | - 0 0 | 0B | 0 0B | 0 0B | 0 0B | 0B | 0 0 | 0B 0B
4 | 0 0B 0B 0 | - 0 0 | 0B | 0 0B | 0 0B | 0 0B | 0B | 0 0 | 0B 0B
5 | 0 0B 0B 0 | - 0 0 | 0B | 0 0B | 0 0B | 0 0B | 0B | 0 0 | 0B 0B
6 | 1 932B 0B 0 | - 0.00 0.00 | 1.8KB | 0 0B | 0 0B | 1 932B | 1.8KB | 1 0.51 | 232B 0B
total | 1 932B 0B 0 | - - - | 156B | 0 0B | 0 0B | 3 2.8KB | 1.8KB | 1 21.8 | 232B 0B
total | 1 932B 0B 0 | - - - | 156B | 0 0B | 0 0B | 3 2.8KB | 1.8KB | 1 20.2 | 232B 0B
--------------------------------------------------------------------------------------------------------------------------------------------
WAL: 1 files (0B) in: 126B written: 156B (24% overhead)
Flushes: 2
Expand Down
24 changes: 12 additions & 12 deletions testdata/metrics
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -350,14 +350,14 @@ metrics
| | | | ingested | moved | written | | amp | val sep
level | tables size val-bl vtables | score ff cff | in | tables size | tables size | tables size | read | r w | refsz valblk
------+-----------------------------+----------------+-------+--------------+--------------+--------------+-------+----------+--------------
0 | 7 6.0KB 0B 0 | - 0.25 0.25 | 165B | 0 0B | 0 0B | 9 7.5KB | 0B | 1 54.2 | 644B 0B
0 | 7 6.0KB 0B 0 | - 0.25 0.25 | 165B | 0 0B | 0 0B | 9 7.5KB | 0B | 1 50.3 | 644B 0B
1 | 0 0B 0B 0 | - 0 0 | 0B | 0 0B | 0 0B | 0 0B | 0B | 0 0 | 0B 0B
2 | 0 0B 0B 0 | - 0 0 | 0B | 0 0B | 0 0B | 0 0B | 0B | 0 0 | 0B 0B
3 | 0 0B 0B 0 | - 0 0 | 0B | 0 0B | 0 0B | 0 0B | 0B | 0 0 | 0B 0B
4 | 0 0B 0B 0 | - 0 0 | 0B | 0 0B | 0 0B | 0 0B | 0B | 0 0 | 0B 0B
5 | 0 0B 0B 0 | - 0 0 | 0B | 0 0B | 0 0B | 0 0B | 0B | 0 0 | 0B 0B
6 | 2 1.5KB 0B 0 | - 0.00 0.00 | 1.5KB | 0 0B | 0 0B | 2 1.5KB | 1.5KB | 1 0.99 | 0B 0B
total | 9 7.5KB 0B 0 | - - - | 165B | 0 0B | 0 0B | 11 9.1KB | 1.5KB | 2 64.4 | 644B 0B
total | 9 7.5KB 0B 0 | - - - | 165B | 0 0B | 0 0B | 11 9.1KB | 1.5KB | 2 60.5 | 644B 0B
--------------------------------------------------------------------------------------------------------------------------------------------
WAL: 1 files (0B) in: 116B written: 165B (42% overhead)
Flushes: 3
Expand Down Expand Up @@ -422,14 +422,14 @@ metrics
| | | | ingested | moved | written | | amp | val sep
level | tables size val-bl vtables | score ff cff | in | tables size | tables size | tables size | read | r w | refsz valblk
------+-----------------------------+----------------+-------+--------------+--------------+--------------+-------+----------+--------------
0 | 0 0B 0B 0 | - 0 0 | 165B | 0 0B | 0 0B | 9 7.5KB | 0B | 0 54.2 | 0B 0B
0 | 0 0B 0B 0 | - 0 0 | 165B | 0 0B | 0 0B | 9 7.5KB | 0B | 0 50.3 | 0B 0B
1 | 0 0B 0B 0 | - 0 0 | 0B | 0 0B | 0 0B | 0 0B | 0B | 0 0 | 0B 0B
2 | 0 0B 0B 0 | - 0 0 | 0B | 0 0B | 0 0B | 0 0B | 0B | 0 0 | 0B 0B
3 | 0 0B 0B 0 | - 0 0 | 0B | 0 0B | 0 0B | 0 0B | 0B | 0 0 | 0B 0B
4 | 0 0B 0B 0 | - 0 0 | 0B | 0 0B | 0 0B | 0 0B | 0B | 0 0 | 0B 0B
5 | 0 0B 0B 0 | - 0 0 | 0B | 0 0B | 0 0B | 0 0B | 0B | 0 0 | 0B 0B
6 | 9 7.4KB 0B 0 | - 0.00 0.00 | 7.5KB | 0 0B | 0 0B | 9 7.4KB | 7.5KB | 1 1.00 | 644B 0B
total | 9 7.4KB 0B 0 | - - - | 165B | 0 0B | 0 0B | 18 15KB | 7.5KB | 1 101 | 644B 0B
total | 9 7.4KB 0B 0 | - - - | 165B | 0 0B | 0 0B | 18 15KB | 7.5KB | 1 97.5 | 644B 0B
--------------------------------------------------------------------------------------------------------------------------------------------
WAL: 1 files (0B) in: 116B written: 165B (42% overhead)
Flushes: 3
Expand Down Expand Up @@ -545,14 +545,14 @@ metrics
| | | | ingested | moved | written | | amp | val sep
level | tables size val-bl vtables | score ff cff | in | tables size | tables size | tables size | read | r w | refsz valblk
------+-----------------------------+----------------+-------+--------------+--------------+--------------+-------+----------+--------------
0 | 6 4.5KB 0B 0 | - 0.50 0.50 | 211B | 3 2.2KB | 0 0B | 12 9.7KB | 0B | 2 53.3 | 0B 0B
0 | 6 4.5KB 0B 0 | - 0.50 0.50 | 211B | 3 2.2KB | 0 0B | 12 9.7KB | 0B | 2 50.2 | 0B 0B
1 | 0 0B 0B 0 | - 0 0 | 0B | 0 0B | 0 0B | 0 0B | 0B | 0 0 | 0B 0B
2 | 0 0B 0B 0 | - 0 0 | 0B | 0 0B | 0 0B | 0 0B | 0B | 0 0 | 0B 0B
3 | 0 0B 0B 0 | - 0 0 | 0B | 0 0B | 0 0B | 0 0B | 0B | 0 0 | 0B 0B
4 | 0 0B 0B 0 | - 0 0 | 0B | 0 0B | 0 0B | 0 0B | 0B | 0 0 | 0B 0B
5 | 0 0B 0B 0 | - 0 0 | 0B | 0 0B | 0 0B | 0 0B | 0B | 0 0 | 0B 0B
6 | 9 7.4KB 0B 0 | - 0.00 0.00 | 7.5KB | 0 0B | 0 0B | 9 7.4KB | 7.5KB | 1 1.00 | 644B 0B
total | 15 12KB 0B 0 | - - - | 2.4KB | 3 2.2KB | 0 0B | 21 20KB | 7.5KB | 3 8.53 | 644B 0B
total | 15 12KB 0B 0 | - - - | 2.4KB | 3 2.2KB | 0 0B | 21 20KB | 7.5KB | 3 8.27 | 644B 0B
--------------------------------------------------------------------------------------------------------------------------------------------
WAL: 1 files (0B) in: 176B written: 211B (20% overhead)
Flushes: 8
Expand Down Expand Up @@ -630,14 +630,14 @@ metrics
| | | | ingested | moved | written | | amp | val sep
level | tables size val-bl vtables | score ff cff | in | tables size | tables size | tables size | read | r w | refsz valblk
------+-----------------------------+----------------+-------+--------------+--------------+--------------+-------+----------+--------------
0 | 13 9.8KB 0B 0 | - 0.50 0.50 | 277B | 3 2.2KB | 0 0B | 19 15KB | 0B | 2 60.0 | 0B 0B
0 | 13 9.8KB 0B 0 | - 0.50 0.50 | 277B | 3 2.2KB | 0 0B | 19 15KB | 0B | 2 57.7 | 0B 0B
1 | 0 0B 0B 0 | - 0 0 | 0B | 0 0B | 0 0B | 0 0B | 0B | 0 0 | 0B 0B
2 | 0 0B 0B 0 | - 0 0 | 0B | 0 0B | 0 0B | 0 0B | 0B | 0 0 | 0B 0B
3 | 0 0B 0B 0 | - 0 0 | 0B | 0 0B | 0 0B | 0 0B | 0B | 0 0 | 0B 0B
4 | 0 0B 0B 0 | - 0 0 | 0B | 0 0B | 0 0B | 0 0B | 0B | 0 0 | 0B 0B
5 | 0 0B 0B 0 | - 0 0 | 0B | 0 0B | 0 0B | 0 0B | 0B | 0 0 | 0B 0B
6 | 9 7.4KB 0B 0 | - 0.00 0.00 | 7.5KB | 0 0B | 0 0B | 9 7.4KB | 7.5KB | 1 1.00 | 644B 0B
total | 22 17KB 0B 0 | - - - | 2.5KB | 3 2.2KB | 0 0B | 28 25KB | 7.5KB | 3 10.4 | 644B 0B
total | 22 17KB 0B 0 | - - - | 2.5KB | 3 2.2KB | 0 0B | 28 25KB | 7.5KB | 3 10.2 | 644B 0B
--------------------------------------------------------------------------------------------------------------------------------------------
WAL: 1 files (0B) in: 223B written: 277B (24% overhead)
Flushes: 9
Expand Down Expand Up @@ -727,14 +727,14 @@ metrics zero-cache-hits-misses
| | | | ingested | moved | written | | amp | val sep
level | tables size val-bl vtables | score ff cff | in | tables size | tables size | tables size | read | r w | refsz valblk
------+-----------------------------+----------------+-------+--------------+--------------+--------------+-------+----------+--------------
0 | 11 8.2KB 0B 0 | - 0.50 0.50 | 277B | 3 2.2KB | 0 0B | 19 15KB | 0B | 2 60.0 | 0B 0B
0 | 11 8.2KB 0B 0 | - 0.50 0.50 | 277B | 3 2.2KB | 0 0B | 19 15KB | 0B | 2 57.7 | 0B 0B
1 | 0 0B 0B 0 | - 0 0 | 0B | 0 0B | 0 0B | 0 0B | 0B | 0 0 | 0B 0B
2 | 0 0B 0B 0 | - 0 0 | 0B | 0 0B | 0 0B | 0 0B | 0B | 0 0 | 0B 0B
3 | 0 0B 0B 0 | - 0 0 | 0B | 0 0B | 0 0B | 0 0B | 0B | 0 0 | 0B 0B
4 | 0 0B 0B 0 | - 0 0 | 0B | 0 0B | 0 0B | 0 0B | 0B | 0 0 | 0B 0B
5 | 0 0B 0B 0 | - 0 0 | 0B | 0 0B | 0 0B | 0 0B | 0B | 0 0 | 0B 0B
6 | 10 8.2KB 0B 0 | - 0.00 0.00 | 7.5KB | 1 765B | 0 0B | 9 7.4KB | 7.5KB | 1 1.00 | 644B 0B
total | 21 16KB 0B 0 | - - - | 3.3KB | 4 3.0KB | 0 0B | 28 26KB | 7.5KB | 3 8.27 | 644B 0B
total | 21 16KB 0B 0 | - - - | 3.3KB | 4 3.0KB | 0 0B | 28 26KB | 7.5KB | 3 8.08 | 644B 0B
--------------------------------------------------------------------------------------------------------------------------------------------
WAL: 1 files (0B) in: 223B written: 277B (24% overhead)
Flushes: 9
Expand Down Expand Up @@ -868,14 +868,14 @@ metrics zero-cache-hits-misses
| | | | ingested | moved | written | | amp | val sep
level | tables size val-bl vtables | score ff cff | in | tables size | tables size | tables size | read | r w | refsz valblk
------+-----------------------------+----------------+-------+--------------+--------------+--------------+-------+----------+--------------
0 | 0 0B 0B 0 | - 0 0 | 277B | 3 2.2KB | 0 0B | 19 15KB | 0B | 0 60.0 | 0B 0B
0 | 0 0B 0B 0 | - 0 0 | 277B | 3 2.2KB | 0 0B | 19 15KB | 0B | 0 57.7 | 0B 0B
1 | 0 0B 0B 0 | - 0 0 | 0B | 0 0B | 0 0B | 0 0B | 0B | 0 0 | 0B 0B
2 | 0 0B 0B 0 | - 0 0 | 0B | 0 0B | 0 0B | 0 0B | 0B | 0 0 | 0B 0B
3 | 0 0B 0B 0 | - 0 0 | 0B | 0 0B | 0 0B | 0 0B | 0B | 0 0 | 0B 0B
4 | 0 0B 0B 0 | - 0 0 | 0B | 0 0B | 0 0B | 0 0B | 0B | 0 0 | 0B 0B
5 | 0 0B 0B 0 | - 0 0 | 0B | 0 0B | 0 0B | 0 0B | 0B | 0 0 | 0B 0B
6 | 18 14KB 0B 0 | - 0.00 0.00 | 15KB | 2 1.5KB | 0 0B | 16 13KB | 15KB | 1 0.85 | 644B 0B
total | 18 14KB 0B 0 | - - - | 4.0KB | 5 3.7KB | 0 0B | 35 32KB | 15KB | 1 8.23 | 644B 0B
total | 18 14KB 0B 0 | - - - | 4.0KB | 5 3.7KB | 0 0B | 35 32KB | 15KB | 1 8.07 | 644B 0B
--------------------------------------------------------------------------------------------------------------------------------------------
WAL: 1 files (0B) in: 223B written: 277B (24% overhead)
Flushes: 9
Expand Down