fix(telemetry): report tracesUrl and explain trace disconnects

[RamanKharchee]

Summary

Two related telemetry gaps: the agent never reported tracesUrl, and never explained why traces were disconnected.

1. tracesUrl was always empty for otel_clickhouse clusters. The legacy sink passed clickhouse_url (CLICKHOUSE_HOST) into get_trace_url() (nudgebee_sink.py:1381). The Go port hardcoded that path to "", with a comment claiming "we don't run a local ClickHouse, so it's always ''" — stale, since main.go does read and probe CLICKHOUSE_HOST.

That left agent_service.go:235's TracesUrl != nil gate permanently closed, so TraceProviderConfig["otel_traces"] was never written and Last9 clusters resolved table otel_traces instead of otel.traces. tool_trace_clickhouse.go:136 missed the same last9.io check. Standard ClickHouse was unaffected — GetTracesTableNames defaults to the same value.

Adds isClickHouseEnabled as the otel_clickhouse counterpart of isJaegerEnabled, and reports the host.

2. tracesConnectionError was declared and rendered, but emitted by nobody. Declared at agent_service.go:94, read at agentHealth.jsx:198, rendered by renderReason(!isTracesManagerConnected, tracesError). Since renderReason needs disconnected && error and the value was permanently '', the "Reason - ..." line under a Disconnected Traces pill has never once rendered. probeClickhouse now returns a reason alongside its status.

Condition	tracesConnectionError
ClickHouse healthy	""
Ping fails	ClickHouse ping failed at ch.svc:8123: connection refused
CLICKHOUSE_HOST unset	CLICKHOUSE_HOST is not set: no traces backend is configured
TRACES_ENABLED=false	"" — off on purpose isn't a failure
Traces up via any provider	""

No backend or UI change needed; both consumers already exist.

Type of change

• Bug fix (non-breaking)

Chart version

• No version bump needed (docs/CI only)

Runner-only Go change, no chart templates touched. The closest precedent — #517, which also added a telemetry field — didn't bump either. Shout if that's wrong and I'll add one.

Test plan

• go build ./..., go vet, and the full agent suite pass (relevant: the httpProbe split below touches every probe caller)
• New unit tests: probe reason matrix, redactUserinfo, reason-clears-on-recovery, and a marshalling test pinning the no-omitempty contract
• Not installed on a real cluster — verification is build + tests only

helm lint / ct lint / helm template not applicable (no chart changes).

Review Notes → Risks & Counterarguments

httpProbe refactor touches every probe caller. Split into a thin bool wrapper over a new httpProbeErr that preserves the error, so Grafana/OpenCost/logs callers are untouched by construction. Full suite passes. Behaviour is identical, but it's the widest-blast-radius hunk here.

Two deliberate deviations from legacy parity, both documented in-code:

• ClickHouse is checked last in traceURL, not first. The legacy order returns the ClickHouse host even when TRACE_TABLE makes the provider bigquery, and the backend then backtick-quotes that host as a BigQuery table name.
• isClickHouseEnabled is deliberately not used by traceStatus. Gating tracesEnabled on the URL would turn traces off for TRACES_ENABLED=true-without-host — a config main.go:1288 explicitly supports for external ClickHouse the agent can't probe.

The no-omitempty on tracesConnectionError is load-bearing. The collector merges into connection_status with jsonb || (telemetry_handler.py:267), so keys are sticky. An omitted key would strand a stale reason in the DB forever, resurfacing under any later failure that produces no message. A recovered ClickHouse must post an explicit "" to clear it. There's a marshalling test guarding this, because it's exactly what a future "add omitempty" cleanup would silently break.

Newly-activated backend branch. Populating tracesUrl makes agent_service.go:235 pass its gate for the first time and start writing TraceProviderConfig["otel_traces"]. For standard ClickHouse the written value equals the existing default, so no behaviour change; Last9 correctly switches to otel.traces. I traced the consumers but haven't exercised this against a live agent.

Credential exposure was a real risk here. The reason string lands in connection_status JSON and renders verbatim in the UI, and CLICKHOUSE_HOST may be a full URL. redactUserinfo strips user:pass@ and probeCause unwraps *url.Error so the request URL never reaches the string. Belt-and-braces — net/http already strips passwords — but the username survived that.

Pre-existing bug, deliberately not fixed: probeClickhouse builds fmt.Sprintf("http://%s:%s/ping", host, port), which breaks for a URL-form CLICKHOUSE_HOST (http://https://x:8123/ping); pkg/clickhouse.normalizeHost exists precisely for that. Those clusters already reported traces disconnected — the change is that they now get a "ping failed" reason that's true but points at the wrong culprit. In practice Last9-style setups sidestep it via TRACES_ENABLED=true. Worth a follow-up routing the probe through normalizeHost.

Related issues

Fixes nudgebee/nudgebee-enterprise#34231
Refs nudgebee/nudgebee-enterprise#34172 — possibly related (Traces page null for clickhouse otel), deliberately not linked as a fix. This only changes real behaviour for Last9 hosts, so it's unlikely to be that root cause unless that env is Last9-backed.

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request enhances the ClickHouse probing mechanism to capture and report specific connection errors when traces are down, propagating them to the telemetry payload. The feedback highlights a potential bug where full URLs in CLICKHOUSE_HOST could result in malformed probe URLs, and suggests robust URL parsing to handle this. Additionally, it is recommended to fully drain the HTTP response body in httpProbeErr to prevent connection leaks, along with adding the necessary imports.

[github-actions]

📦 Image Tags Updated

I've automatically updated the image tags in `charts/nudgebee-agent/values.yaml` to the latest versions from GHCR for the `main` branch.

The image tags are now synchronized with the latest builds and ready for release.