Skip to content

fix(metrics): correct storage backend label and doubled _total suffix#134

Merged
sionsmith merged 2 commits into
mainfrom
fix/metrics-backend-label-and-counter-names
Jul 21, 2026
Merged

fix(metrics): correct storage backend label and doubled _total suffix#134
sionsmith merged 2 commits into
mainfrom
fix/metrics-backend-label-and-counter-names

Conversation

@sionsmith

Copy link
Copy Markdown
Contributor

Summary

Fixes the two metric defects reported in strimzi-backup-operator#50:

  • backend="filesystem" on S3 backupsSegmentFlusher::write hardcoded "filesystem" when recording kafka_backup_storage_write_bytes_total and kafka_backup_storage_write_latency_seconds, regardless of the configured backend. StorageBackend now exposes a backend_name() method (s3, azure, gcs, filesystem, memory) and the flusher labels metrics with the backend it actually wrote to. To be clear for the reporter: segments were always streamed directly to object storage — only the label was wrong, no PVC is needed.
  • Doubled _total_total suffix — counters were registered with an explicit _total suffix, but prometheus-client appends _total to counters at encode time, so series were exposed as e.g. kafka_backup_records_total_total / kafka_backup_storage_write_bytes_total_total. Counters are now registered un-suffixed, so exposed series finally match the names documented in docs/configuration.md (kafka_backup_records_total, kafka_backup_storage_write_bytes_total, …). status-watch continues to accept the old doubled names so it can watch pre-0.15.12 endpoints.

Breaking (dashboards only): any dashboard/alert built on the doubled *_total_total names must move to the documented single-_total names.

The third symptom in that issue — static kafka_backup_lag_records / kafka_backup_lag_records_sum / kafka_backup_snapshot_records_remaining — is the cardinality-budget freeze already fixed in v0.15.11 (#131): the reporter's long-running job predates the v0.15.11 image rollout. Verified below that v0.15.11+ gauges track progress correctly.

Verification

Reproduced locally against apache/kafka:3.7.1 + MinIO with 3.2M records (~1.9GB) in a 6-partition topic, snapshot mode (stop_at_current_offsets: true), scraping /metrics every 400ms during the run:

v0.15.10 image behaviour this branch
kafka_backup_lag_records frozen at initial values for entire run counts down to 0 per partition
kafka_backup_snapshot_records_remaining absent 3,191,285 → 0
kafka_backup_lag_records_sum absent 3,191,285 → 0
storage write bytes kafka_backup_storage_write_bytes_total_total{backend="filesystem"} kafka_backup_storage_write_bytes_total{backend="s3"}
  • cargo test --workspace — 259 tests pass, including new regression tests:
    • counters_expose_single_total_suffix (no _total_total anywhere in encoded output)
    • test_segment_flush_labels_actual_backend (flusher labels the real backend, never a hardcoded filesystem)
  • cargo clippy --workspace --all-targets -- -D warnings clean

Follow-ups

  • kafka-backup-docs reference/metrics.md documents the _total_total names and should be updated to the single-_total form once this ships.
  • strimzi-backup-operator DEFAULT_BACKUP_IMAGE should be bumped to v0.15.12 after release.

🤖 Generated with Claude Code

sionsmith and others added 2 commits July 21, 2026 20:33
…bled _total suffix

Segment uploads recorded storage write metrics with a hardcoded
backend="filesystem" label even when writing to S3/Azure/GCS. Expose the
actual backend via a StorageBackend::backend_name() method and use it in
the segment flusher.

Counters were registered with an explicit _total suffix, but
prometheus-client appends _total to counters at encode time, so series
were exposed as e.g. kafka_backup_records_total_total. Register the
un-suffixed names so exposed series match the documented
kafka_backup_*_total form. status-watch keeps accepting the old doubled
names so it can still watch pre-0.15.12 endpoints.

Addresses osodevops/strimzi-backup-operator#50.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@sionsmith
sionsmith merged commit ec2995a into main Jul 21, 2026
20 checks passed
@sionsmith
sionsmith deleted the fix/metrics-backend-label-and-counter-names branch July 21, 2026 19:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant