telemetry: materialize dropped-event counter children at 0#927
telemetry: materialize dropped-event counter children at 0#927philosaether wants to merge 3 commits into
Conversation
A labeled Prometheus counter exports no series until its first labels() call, so telemetry_events_dropped stayed invisible until an event was actually dropped — impossible to alert on or graph, and "no drops" was indistinguishable from "metric missing / scrape broken". Pre-create the (namespace, reason) children at 0 on emitter start, for each reason the emitter can emit, so the metric is always present. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
WalkthroughPrometheus metrics now pre-materialize bounded label series during API, deriver, and emitter startup. Reconciliation reports pending message embeddings through a gauge, with tests covering valid domains, process isolation, dropped counters, and disabled metrics. ChangesTelemetry metrics initialization
Estimated code review effort: 4 (Complex) | ~45 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/telemetry/prometheus/metrics.py`:
- Around line 341-347: Update initialize_telemetry_dropped_metrics to catch only
the expected Prometheus labels(reason=reason) validation exception, using the
appropriate specific Prometheus exception type. Let unexpected exceptions from
telemetry_events_dropped_counter.labels propagate instead of routing them
through _handle_metric_error, preserving the existing handling for valid label
errors.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: f114054c-c059-4b97-816c-ba9cad4842e6
📒 Files selected for processing (2)
src/telemetry/emitter.pysrc/telemetry/prometheus/metrics.py
| for reason in reasons: | ||
| try: | ||
| telemetry_events_dropped_counter.labels(reason=reason) | ||
| except Exception as e: | ||
| self._handle_metric_error( | ||
| "initialize_telemetry_dropped_metrics", e | ||
| ) |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Locate metrics.py:"
fd -a 'metrics.py$' . | sed 's#^\./##'
echo
echo "File size and relevant section:"
wc -l src/telemetry/prometheus/metrics.py
sed -n '300,370p' src/telemetry/prometheus/metrics.py
echo
echo "Search for _handle_metric_error and telemetry_events_dropped_counter definitions/usages:"
rg -n "_handle_metric_error|telemetry_events_dropped_counter|initialized_telemetry_dropped|class .*Counter|def labels" src/telemetry src 2>/dev/null | head -200Repository: plastic-labs/honcho
Length of output: 7997
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Locate metrics.py:"
fd -a 'metrics.py$' . | sed 's#^\./##'
echo
echo "File size and relevant section:"
wc -l src/telemetry/prometheus/metrics.py
sed -n '300,370p' src/telemetry/prometheus/metrics.py
echo
echo "Search for _handle_metric_error and telemetry_events_dropped_counter definitions/usages:"
rg -n "_handle_metric_error|telemetry_events_dropped_counter|initialized_telemetry_dropped|class .*Counter|def labels" src/telemetry/src 2>/dev/null | head -200Repository: plastic-labs/honcho
Length of output: 3365
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Relevant metrics.py sections:"
sed -n '1,90p' src/telemetry/prometheus/metrics.py
sed -n '130,190p' src/telemetry/prometheus/metrics.py
echo
echo "Inspect prometheus module requirements/imports:"
sed -n '1,60p' src/telemetry/prometheus/metrics.py | grep -n "from\|import\|prometheus" || true
echo
echo "Relevant test references (if any):"
sed -n '1,220p' tests/integration/test_token_metrics.py | grep -n "namespaced\|labels\|initialize_telemetry_dropped_metrics\|Prometheus" -i || trueRepository: plastic-labs/honcho
Length of output: 6370
🌐 Web query:
prometheus_client Counter.labels InvalidLabelValueError ValueError documentation
💡 Result:
In the prometheus_client library, a ValueError associated with.labels typically indicates an attempt to provide incorrect or malformed label data [1]. While there is no single error named "InvalidLabelValueError", the library raises standard ValueError exceptions with descriptive messages when label constraints are violated [1]. Common scenarios that trigger a ValueError when calling.labels include: 1. Incorrect Label Count: Providing a different number of arguments in.labels than the number of label names defined during the initialization of the metric [1]. 2. Mismatched Keyword Arguments: Providing keyword arguments that do not match the declared label names [1]. 3. Mixing Arguments and Keywords: Passing both positional arguments and keyword arguments to.labels [1]. 4. Calling Labels on an Already Labeled Metric: Attempting to call.labels on a metric instance that already has specific label values assigned (e.g., trying to chain.labels calls) [1]. 5. Missing Label Definition: Attempting to call.labels on a metric that was not initialized with any label names [1]. Additionally, if you receive an error stating that a metric is missing label values (often seen when attempting to call.inc or.set directly on a metric that requires labels), it is because you must first call.labels(...) to obtain a specific child metric instance before performing operations [2][3]. For example: Correct usage: counter = Counter('my_metric', 'help', ['method']) counter.labels(method='get').inc # Correct Incorrect usage (results in error): counter = Counter('my_metric', 'help', ['method']) counter.inc # ValueError or AttributeError: missing label values or _value attribute Always ensure that the arguments passed to.labels strictly align with the label names specified in the constructor [1].
Citations:
- 1: https://github.com/prometheus/client_python/blob/master/prometheus_client/metrics.py
- 2: 'Counter' object has no attribute '_value' prometheus/client_python#897
- 3: Raise the correct exception if the metrics are not observable prometheus/client_python#666
Catch only expected Prometheus labels(...) validation errors.
initialize_telemetry_dropped_metrics() catches all Exceptions, so unexpected failures in NamespacedCounter.labels(...) are logged as metric recording failures and startup can continue without the new reason series. Catch Prometheus label/validation errors explicitly, or let unexpected exceptions propagate when metric initialization is critical for alerting.
[stblability_and_availability]
🧰 Tools
🪛 Ruff (0.15.21)
[warning] 344-344: Do not catch blind exception: Exception
(BLE001)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/telemetry/prometheus/metrics.py` around lines 341 - 347, Update
initialize_telemetry_dropped_metrics to catch only the expected Prometheus
labels(reason=reason) validation exception, using the appropriate specific
Prometheus exception type. Let unexpected exceptions from
telemetry_events_dropped_counter.labels propagate instead of routing them
through _handle_metric_error, preserving the existing handling for valid label
errors.
Sources: Coding guidelines, Linters/SAST tools
| except Exception as e: | ||
| self._handle_metric_error("record_telemetry_event_dropped", e) | ||
|
|
||
| def initialize_telemetry_dropped_metrics(self, *, reasons: list[str]) -> None: |
There was a problem hiding this comment.
nice - we should generalize this method to be initialize_counter_series (or something similar) to support initializing other counters in the future
Rajat-Ahuja1997
left a comment
Rajat-Ahuja1997
left a comment
There was a problem hiding this comment.
looks good, 1 comment
Extends #927 (which zero-inited telemetry_events_dropped) to every counter whose label domain is bounded and known at startup, so metrics are present in Prometheus before their first event — a missing series then signals a broken scrape rather than "nothing happened yet". - add initialize_bounded_metrics(instance_type) on PrometheusMetrics; call it per-process from main.py (api) and deriver/__main__.py (deriver). - extract a shared _touch() helper; refactor initialize_telemetry_dropped_metrics onto it (that one stays per-emitter in start() — it's prefix-dependent). - explicit ALL_EVENT_TYPES / HIGH_VOLUME_EVENT_TYPES registry in telemetry.events, drift-guarded by tests that walk BaseEvent subclasses. - only VALID (task_type, token_type, component) tuples for deriver_tokens (the cartesian product would fabricate impossible always-0 series); only high-volume event types for sampled_out; high-cardinality labels (endpoint, workspace_name) left open. - gauges: zero-init embed_now_tasks_in_flight + telemetry_buffer_size; add a new message_embeddings_pending backlog gauge, set each reconciliation cycle and zero-inited at deriver startup (Rajat's pending/in-flight ask). - backfills the tests #927 shipped without. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/telemetry/prometheus/metrics.py`:
- Around line 403-458: Replace the direct gauge initialization calls in
initialize_bounded_metrics with the class’s fail-soft _touch path, including
telemetry_buffer_size_gauge, embed_now_tasks_in_flight_gauge, and
message_embeddings_pending_gauge. Preserve initializing each gauge to zero while
ensuring any metric setup failure is handled consistently without interrupting
startup.
In `@tests/telemetry/test_metric_zero_init.py`:
- Around line 227-241: The test test_api_init_does_not_touch_deriver_counters
currently covers only API-to-deriver isolation. Add a before/after assertion
around prometheus_metrics.initialize_bounded_metrics(instance_type="deriver")
using an API-only bounded metric and its appropriate labels, asserting the
samples are unchanged so deriver initialization cannot materialize API metrics.
- Around line 95-102: Update the validation around _DERIVER_TOKEN_COMBOS to
compare the complete set of token/component pairs against the exact expected
pair set, rather than checking token and component membership independently.
Also assert the combo collection length equals the expected pair-set length so
duplicate entries are rejected, while preserving the existing coverage checks as
appropriate.
- Around line 31-43: Use unique namespaces in test_metric_zero_init.py for all
global REGISTRY assertions: replace the fixed NS value used by sample() with a
module/test-specific namespace, and introduce a separate unique namespace for
the disabled-metrics check at lines 258-269. Update the relevant metric
configuration and assertions to use these namespaces so materialized children
from other tests cannot affect results.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 605569e1-376d-4abd-a7a8-df494bba2741
📒 Files selected for processing (6)
src/deriver/__main__.pysrc/main.pysrc/reconciler/sync_vectors.pysrc/telemetry/events/__init__.pysrc/telemetry/prometheus/metrics.pytests/telemetry/test_metric_zero_init.py
| NS = "test" | ||
|
|
||
|
|
||
| @pytest.fixture | ||
| def metrics_enabled(monkeypatch: pytest.MonkeyPatch) -> Iterator[None]: | ||
| monkeypatch.setattr("src.config.settings.METRICS.ENABLED", True) | ||
| monkeypatch.setattr("src.config.settings.METRICS.NAMESPACE", NS) | ||
| yield | ||
|
|
||
|
|
||
| def sample(name: str, **labels: str) -> float | None: | ||
| """Value of a series if it exists, else None. Never materializes it.""" | ||
| return REGISTRY.get_sample_value(name, {"namespace": NS, **labels}) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== candidate files =="
git ls-files | rg '(^|/)test_metric_zero_init\.py$|src/telemetry|prometheus_metrics|config/settings|telemetry' || true
echo "== test file size =="
wc -l tests/telemetry/test_metric_zero_init.py 2>/dev/null || true
echo "== relevant imports and fixtures =="
sed -n '1,80p' tests/telemetry/test_metric_zero_init.py 2>/dev/null || true
echo "== relevant body around assertions =="
sed -n '220,290p' tests/telemetry/test_metric_zero_init.py 2>/dev/null || true
echo "== search for REGISTRY and sample definitions/usages =="
rg -n "REGISTRY|def sample|initialize_bounded_metrics|telemetry_events_emitted_total|metrics_enabled" tests src 2>/dev/null || trueRepository: plastic-labs/honcho
Length of output: 12694
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== Prometheus/telemetry module files =="
git ls-files | rg 'telemetry|prometheus_metrics|config/settings' | sed -n '1,200p' || true
echo "== all occurrences of initialize_bounded_metrics =="
rg -n "def initialize_bounded_metrics|initialize_bounded_metrics|METRICS|METRICS_ENABLED|ENABLED" . --glob '!node_modules' --glob '!dist' --glob '!build' 2>/dev/null | sed -n '1,240p' || trueRepository: plastic-labs/honcho
Length of output: 24934
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== relevant Prometheus metrics source =="
sed -n '1,120p' src/telemetry/prometheus/metrics.py
echo "---"
sed -n '340,420p' src/telemetry/prometheus/metrics.py
echo "---"
sed -n '470,535p' src/telemetry/prometheus/metrics.py
echo "== test_metric_zero_init full outline/essential sections =="
sed -n '80,180p' tests/telemetry/test_metric_zero_init.py
echo "---"
sed -n '180,269p' tests/telemetry/test_metric_zero_init.py
echo "== registry cleanup/unregister patterns in test/telemetry files =="
rg -n "\b(unregister(REGISTRY)?|REGISTRY\.(unregister|clear)|get_sample_value|register\()" tests src --glob '!node_modules' 2>/dev/null | sed -n '1,220p'Repository: plastic-labs/honcho
Length of output: 18711
🌐 Web query:
prometheus_client REGISTRY unregister get_sample_value preserves metrics get_sample_value None if not registered
💡 Result:
In the Prometheus Python client library, the behavior of REGISTRY.unregister(collector) and get_sample_value is as follows: REGISTRY.unregister(collector) The unregister(collector) method removes the specified collector from the registry, meaning its metrics will no longer be collected or reported during subsequent scrapes [1][2][3]. Unregistering a collector effectively tells the registry to forget about that source [3]. It does not, however, inherently "reset" or clear the internal state of a metric if it were to be re-registered later, as the metric instances maintain their own state independently of the registry [4][5]. To fully reset metrics, you must either create a new registry or manually clear/reset the metric objects themselves [4][5]. get_sample_value(name, labels=None) The get_sample_value function returns None when a specific metric sample cannot be found in the registry [6][7]. This occurs if: 1. The metric name is not registered [6][7]. 2. The metric is registered but has not yet had any samples recorded for the provided label set [6][7][8]. 3. The labels are passed in an incorrect format (e.g., as a list instead of a dictionary), which can cause the lookup to fail and return None even if the metric exists [6][7]. Because get_sample_value returns None when a sample is missing, it is common practice in testing to use it with a default value (e.g., REGISTRY.get_sample_value(...) or 0) to avoid errors when interacting with metrics that have not yet been initialized or updated [6][7].
Citations:
- 1: https://github.com/prometheus/client_python/blob/master/prometheus_client/registry.py
- 2: http://prometheus.github.io/client_python/registry/
- 3: https://stackoverflow.com/questions/69292067/stop-scraping-of-default-metric-in-prometheus-client-in-python
- 4: How do I reset the prometheus internal state? prometheus/client_python#828
- 5: https://stackoverflow.com/questions/73198616/how-do-i-reset-a-prometheus-python-client-python-runtime-between-pytest-test-fun
- 6: https://stackoverflow.com/questions/56132633/prometheus-registry-get-sample-value-returns-none
- 7: https://www.exchangetuts.com/index.php/prometheus-registryget-sample-value-returns-none-1757379004917116
- 8: Metrics accessed via labels() do not have collect() prometheus/client_python#365
Use unique namespaces for global-registry assertions. REGISTRY retains metric children after they are materialized, so fixed "test" and "disabled_ns" labels can make later assertions reflect materialization from another test instead of this initializer. Generate a module/test-specific namespace for sample() and a separate disabled namespace for the disabled-metrics check.
📍 Affects 1 file
tests/telemetry/test_metric_zero_init.py#L31-L43(this comment)tests/telemetry/test_metric_zero_init.py#L258-L269
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/telemetry/test_metric_zero_init.py` around lines 31 - 43, Use unique
namespaces in test_metric_zero_init.py for all global REGISTRY assertions:
replace the fixed NS value used by sample() with a module/test-specific
namespace, and introduce a separate unique namespace for the disabled-metrics
check at lines 258-269. Update the relevant metric configuration and assertions
to use these namespaces so materialized children from other tests cannot affect
results.
| def test_api_init_does_not_touch_deriver_counters(): | ||
| """api-only init must not materialize or change a deriver-only counter. | ||
|
|
||
| Delta-based (before == after) so it's robust to prior tests having | ||
| materialized the series. | ||
| """ | ||
| labels = dict( | ||
| task_type=DeriverTaskTypes.INGESTION.value, | ||
| token_type=TokenTypes.INPUT.value, | ||
| component=DeriverComponents.PROMPT.value, | ||
| ) | ||
| before = sample("deriver_tokens_processed_total", **labels) | ||
| prometheus_metrics.initialize_bounded_metrics(instance_type="api") | ||
| after = sample("deriver_tokens_processed_total", **labels) | ||
| assert before == after |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Test deriver-to-API isolation too.
This only protects the API → deriver direction. Add the inverse before/after assertion around instance_type="deriver" using an API-only bounded series, so a deriver startup regression cannot silently materialize API metrics.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/telemetry/test_metric_zero_init.py` around lines 227 - 241, The test
test_api_init_does_not_touch_deriver_counters currently covers only
API-to-deriver isolation. Add a before/after assertion around
prometheus_metrics.initialize_bounded_metrics(instance_type="deriver") using an
API-only bounded metric and its appropriate labels, asserting the samples are
unchanged so deriver initialization cannot materialize API metrics.
I1: _DERIVER_TOKEN_COMBOS was factored task-independently, materializing the impossible (ingestion, input, previous_summary) series — previous_summary is summary-only. Make combos task-aware (_DERIVER_TOKEN_COMBOS_BY_TASK) so no always-0 impossible series is fabricated, matching the PR's own goal. Tests tightened to assert the ingestion/previous_summary series is absent. I2: the three gauge .set(0) zero-inits were bare while the counter inits go through the fail-soft _touch. Add _set_gauge_zero() so a gauge init can't propagate an exception into process startup either. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
🧹 Nitpick comments (1)
src/telemetry/prometheus/metrics.py (1)
379-381: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueUse a Google-style docstring.
Document the
gaugeparameter in anArgs:section.Proposed fix
def _set_gauge_zero(self, gauge: NamespacedGauge) -> None: - """Materialize a (namespace-only) gauge at 0. Fail-soft, like ``_touch``: - a startup init must never propagate an exception into process boot.""" + """Materialize a namespace-only gauge at zero without failing startup. + + Args: + gauge: The namespace-aware gauge to materialize. + """🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/telemetry/prometheus/metrics.py` around lines 379 - 381, Update the _set_gauge_zero docstring to Google style by adding an Args: section that documents the gauge parameter and its NamespacedGauge type, while preserving the existing behavior description.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@src/telemetry/prometheus/metrics.py`:
- Around line 379-381: Update the _set_gauge_zero docstring to Google style by
adding an Args: section that documents the gauge parameter and its
NamespacedGauge type, while preserving the existing behavior description.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 70ebefb0-e803-4f21-a26c-4130f2563d10
📒 Files selected for processing (2)
src/telemetry/prometheus/metrics.pytests/telemetry/test_metric_zero_init.py
|
Finn-loop review of 313fd41 CI: not configured ( ReviewSummary: PR materializes Prometheus counter children at 0 on startup so metrics like 1. Must fix before merge
2. Should fix soon
3. Safe to mergeNo — merge is mechanically blocked by BLOCKED status, and required CI is not configured. Code quality is high: fail-soft on init (never crashes startup), drift-guard tests prevent stale registries, cartesian-product explosion is avoided by explicit valid-combo tables, and the 294-line test suite is thorough. Label notesThis reviewer's account does not have admin rights to
|
3 participants
What
Pre-create the
telemetry_events_droppedcounter's(namespace, reason)child series at0when the telemetry emitter starts, for eachreasonthe emitter can emit ({prefix}buffer_full,{prefix}send_failed).Why
A labeled Prometheus counter exports no time series until its first
labels(...)call. Sotelemetry_events_droppedwas invisible in Prometheus/Grafana until an event was actually dropped — which means:Materializing the children at
0on startup keeps the metric always present, so a missing series now signals a real problem rather than a healthy system.Changes
src/telemetry/prometheus/metrics.py: addinitialize_telemetry_dropped_metrics(reasons=...), which pre-touches each(namespace, reason)child (namespace auto-injected byNamespacedCounter, matching existingrecord_telemetry_event_droppedusage).src/telemetry/emitter.py: call it on emitter start with the emitter's own prefixed reasons.Testing
Verified in a sandbox: the counter's zero-valued children appear on the metrics endpoint before any drop occurs, and increment correctly when drops happen.
🤖 Generated with Claude Code
Summary by CodeRabbit