fix(metrics): correct storage backend label and doubled _total suffix#134
Merged
Merged
Conversation
…bled _total suffix Segment uploads recorded storage write metrics with a hardcoded backend="filesystem" label even when writing to S3/Azure/GCS. Expose the actual backend via a StorageBackend::backend_name() method and use it in the segment flusher. Counters were registered with an explicit _total suffix, but prometheus-client appends _total to counters at encode time, so series were exposed as e.g. kafka_backup_records_total_total. Register the un-suffixed names so exposed series match the documented kafka_backup_*_total form. status-watch keeps accepting the old doubled names so it can still watch pre-0.15.12 endpoints. Addresses osodevops/strimzi-backup-operator#50. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the two metric defects reported in strimzi-backup-operator#50:
backend="filesystem"on S3 backups —SegmentFlusher::writehardcoded"filesystem"when recordingkafka_backup_storage_write_bytes_totalandkafka_backup_storage_write_latency_seconds, regardless of the configured backend.StorageBackendnow exposes abackend_name()method (s3,azure,gcs,filesystem,memory) and the flusher labels metrics with the backend it actually wrote to. To be clear for the reporter: segments were always streamed directly to object storage — only the label was wrong, no PVC is needed._total_totalsuffix — counters were registered with an explicit_totalsuffix, butprometheus-clientappends_totalto counters at encode time, so series were exposed as e.g.kafka_backup_records_total_total/kafka_backup_storage_write_bytes_total_total. Counters are now registered un-suffixed, so exposed series finally match the names documented indocs/configuration.md(kafka_backup_records_total,kafka_backup_storage_write_bytes_total, …).status-watchcontinues to accept the old doubled names so it can watch pre-0.15.12 endpoints.The third symptom in that issue — static
kafka_backup_lag_records/kafka_backup_lag_records_sum/kafka_backup_snapshot_records_remaining— is the cardinality-budget freeze already fixed in v0.15.11 (#131): the reporter's long-running job predates the v0.15.11 image rollout. Verified below that v0.15.11+ gauges track progress correctly.Verification
Reproduced locally against
apache/kafka:3.7.1+ MinIO with 3.2M records (~1.9GB) in a 6-partition topic, snapshot mode (stop_at_current_offsets: true), scraping/metricsevery 400ms during the run:kafka_backup_lag_recordskafka_backup_snapshot_records_remainingkafka_backup_lag_records_sumkafka_backup_storage_write_bytes_total_total{backend="filesystem"}kafka_backup_storage_write_bytes_total{backend="s3"}cargo test --workspace— 259 tests pass, including new regression tests:counters_expose_single_total_suffix(no_total_totalanywhere in encoded output)test_segment_flush_labels_actual_backend(flusher labels the real backend, never a hardcodedfilesystem)cargo clippy --workspace --all-targets -- -D warningscleanFollow-ups
reference/metrics.mddocuments the_total_totalnames and should be updated to the single-_totalform once this ships.DEFAULT_BACKUP_IMAGEshould be bumped tov0.15.12after release.🤖 Generated with Claude Code