Skip to content

fix(agent): stop light-action 401s for unconfigured proxy datasources - #474

Merged
mayankpande88 merged 3 commits into
mainfrom
fix/jaeger-light-action-allowlist
Jun 11, 2026
Merged

fix(agent): stop light-action 401s for unconfigured proxy datasources#474
mayankpande88 merged 3 commits into
mainfrom
fix/jaeger-light-action-allowlist

Conversation

@mayankpande88

Copy link
Copy Markdown
Contributor

Problem

Loki showed ~all light-action failures over the last 3 days were jaeger:
jaeger_query_traces (368) and jaeger_query_services (190), rejected with:

auth: action "jaeger_query_traces" not in light-action allowlist

Root cause is a class of bug, not just jaeger. The agent only registers a
datasource's actions — both the handler and the light-action allowlist
entry — when that datasource's URL is set at startup (main.go). But the
backend decides what to dispatch from each account's integration config
(e.g. traces=jaeger-via-agent), independent of the agent's env. So any
datasource the backend can select but the chart doesn't wire gets a 401.

The nudgebee-agent chart wires no jaeger URL at all, yet accounts configured
for jaeger-via-agent still dispatch jaeger_query_* → every one rejected.

Same latent risk for chronosphere, pinot, gke — they just aren't in use yet.

Fix (whole class, not just jaeger)

  • registerProxy() always adds a read-only proxy datasource's action names
    to the allowlist (sourced from the package's own Handlers() map, so names
    can't drift). URL set → real handler; URL unset → a stub returning a clear
    "<datasource> not configured" error instead of an auth rejection. Applied to
    jaeger, chronosphere, pinot, signoz, elasticsearch, gcp, http-proxy. Mirrors
    the existing query_data / api_traces_enricher_v2 unconditional-register
    precedent. Prometheus/Loki left as-is (wired by default + auto-discovered).
  • config: JAEGER_URL falls back to JAEGER_QUERY_URL, aligning handler
    registration with the var the telemetry heartbeat already reports.
  • chart: first-class values for the optional trace datasources
    (runner.jaeger.queryUrl, runner.chronosphere.{url,apiKey},
    runner.pinot.{url,authToken,username,password}), surfaced as env in the
    runner ConfigMap. This is what makes jaeger actually return traces — the
    stub only stops the auth failure.

Behavior change

Dispatcher path for a dispatched-but-unconfigured datasource action goes from
401 "not in light-action allowlist" (auth reject) to 200 when
configured, or 500 with an actionable "not configured" message when not.

To resolve the affected tenant

Set runner.jaeger.queryUrl to the in-cluster jaeger-query endpoint and
restart the pod. Without it, jaeger goes 401 → "not configured" 500 — better,
but no traces until the URL is provided.

Validation

  • gofmt / go vet clean, go build ./... OK, agent test suites pass.
  • helm lint passes; helm template renders the new env when set and omits it by default.

The agent only registered a datasource's actions (handler + light-action
allowlist entry) when its URL was set at startup. But the backend dispatches
these actions from per-account integration config, independent of the agent's
env — so any datasource the backend selects but the chart doesn't wire is
rejected with `auth: action "..." not in light-action allowlist` (401).

This surfaced as ~all light-action failures being jaeger_query_traces /
jaeger_query_services: the chart wires no jaeger URL, yet accounts configured
for traces=jaeger-via-agent still dispatch those actions.

Fix the whole class, not just jaeger:

- registerProxy() now always adds a read-only proxy datasource's action names
  to the allowlist (sourced from the package's own Handlers() map, so no
  drift). URL set -> real handler; URL unset -> a stub returning a clear
  "<datasource> not configured" error instead of an auth rejection. Applied to
  jaeger, chronosphere, pinot, signoz, elasticsearch, gcp, http-proxy. Mirrors
  the existing query_data / api_traces_enricher_v2 unconditional-register
  precedent. Prometheus/Loki are left as-is (wired by default + auto-discovered).

- config: JAEGER_URL falls back to JAEGER_QUERY_URL, aligning handler
  registration with the var the telemetry heartbeat already reports.

- chart: first-class values for the optional trace datasources
  (runner.jaeger.queryUrl, runner.chronosphere.{url,apiKey},
  runner.pinot.{url,authToken,username,password}), surfaced as env in the
  runner ConfigMap. This is what makes jaeger actually return traces; the stub
  only stops the auth failure.
@mayankpande88
mayankpande88 requested a review from a team as a code owner June 11, 2026 08:43
@github-actions

Copy link
Copy Markdown
Contributor

📦 Image Tags Updated

I've automatically updated the image tags in `charts/nudgebee-agent/values.yaml` to the latest versions from GHCR for the `main` branch.

The image tags are now synchronized with the latest builds and ready for release.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a registerProxy helper function in the agent to register read-only proxy datasources unconditionally, returning a clear 'not configured' error when disabled instead of failing authorization. It also updates the Helm chart and agent configuration to support Jaeger, Chronosphere, and Pinot. The feedback suggests using the | quote filter for the newly added environment variables in the Helm template to prevent potential YAML parsing errors or unexpected type coercion.

Comment thread charts/nudgebee-agent/templates/runner.yaml
blue4209211
blue4209211 previously approved these changes Jun 11, 2026
Tokens/passwords/URLs with special chars (#, leading symbols, bool/number-
looking values) can break YAML parsing or get type-coerced when unquoted.
Apply | quote to the jaeger/chronosphere/pinot env, matching GRAFANA_URL.
@github-actions

Copy link
Copy Markdown
Contributor

📦 Image Tags Updated

I've automatically updated the image tags in `charts/nudgebee-agent/values.yaml` to the latest versions from GHCR for the `main` branch.

The image tags are now synchronized with the latest builds and ready for release.

@mayankpande88
mayankpande88 merged commit fc58725 into main Jun 11, 2026
8 checks passed
@mayankpande88
mayankpande88 deleted the fix/jaeger-light-action-allowlist branch June 11, 2026 09:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants