fix: propagate trace context end-to-end for agent Services by syn-zhu · Pull Request #1297 · kagent-dev/kagent

syn-zhu · 2026-02-14T01:01:36Z

Summary

Three fixes to enable end-to-end W3C TraceContext propagation across the controller→agent boundary:

AppProtocol on agent Services — Set appProtocol: kgateway.dev/a2a on the Service port created for each Agent CR so AgentGateway's A2A plugin can discover agent Services directly via protocol matching, rather than proxying through the kagent controller (which drops HTTP headers including traceparent).
W3C TraceContext propagator in Python SDK — Configure the W3C TraceContext propagator in kagent-core tracing setup so agent pods correctly extract incoming traceparent headers and propagate them on outgoing requests.
Trace header propagation in Go controller — The A2A server deserializes incoming HTTP requests into JSON-RPC params, discarding the original HTTP headers. When the controller forwards requests to agent pods via the A2A client, traceparent/tracestate are lost. Fix: capture W3C trace context headers from the incoming request into the Go context in the A2A auth middleware (A2AAuthenticator.Wrap), then inject them into outgoing requests in A2ARequestHandler.

All golden test outputs have been updated to include the new appProtocol field, including agent_with_passthrough (added in #1327).

Incorporates changes from https://github.com/opspawn/kagent/commit/d9f2a3a45a26ec11dd4a9a4cf18e4374a374b03f.

Test plan

Golden tests updated and passing — all testdata/outputs/*.json files include appProtocol: "kgateway.dev/a2a" on Service ports
go test ./internal/httpserver/auth/... passes
Deploy agent, verify kubectl get svc <agent> -o jsonpath='{.spec.ports[0].appProtocol}' returns kgateway.dev/a2a
Verify AgentGateway A2A plugin discovers the Service
Send request with traceparent header through gateway → controller → agent pod, verify trace ID is preserved end-to-end

Closes #1295

🤖 Generated with Claude Code

Copilot

Pull request overview

This PR updates the Agent manifest translation so that the Kubernetes Service created for each Agent CR sets an explicit appProtocol, enabling AgentGateway’s A2A plugin to discover and route directly to agent Services (preserving HTTP headers for distributed tracing).

Changes:

Set spec.ports[0].appProtocol: kgateway.dev/a2a on the per-Agent Service port.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

go/core/internal/controller/translator/agent/adk_api_translator.go

EItanya · 2026-02-16T14:53:15Z

Hey there, thanks for the PR, this is a great idea! You will need to update the goldens as well as sign your commits for us to merge this

syn-zhu · 2026-02-20T20:12:56Z

Hey there, thanks for the PR, this is a great idea! You will need to update the goldens as well as sign your commits for us to merge this

Thanks! Just saw this now, but it looks like the issue was already fixed by https://github.com/opspawn/kagent/commit/d9f2a3a45a26ec11dd4a9a4cf18e4374a374b03f :)

Gonna just close this PR, thanks!

syn-zhu · 2026-02-24T11:01:11Z

Oops @EItanya i realized the commit I linked wasn't actually a merged commit, but rather a branch. I've updated and reopened my PR to address the things you mentioned. Please lmk if there's anything else!

krisztianfekete · 2026-02-24T14:13:33Z

Hi @syn-zhu, Could you please test this locally on your end first? As it's Claude-generated code, a brief manual validation, e.g. such as posting before/after screenshots from a tracing tool would be the minimum step to ensure it's ready for contribution.

syn-zhu · 2026-02-26T04:15:14Z

End-to-End Test Results: Trace Propagation

Tested on a live EKS cluster running kagent v0.7.13 with Langfuse (OTLP-backed) as the trace backend. Each test sends an A2A tasks/send request with a known traceparent header and verifies what Langfuse receives.

Setup

Agent: platform_assistant (Declarative agent with Anthropic via agentgateway-proxy)
LLM path: Agent → agentgateway-proxy → Anthropic API
OTEL path: Agent → gRPC OTLP → otel-collector → HTTP OTLP → Langfuse
Controller: kagent-controller proxies A2A requests to agent pods

Stage 0 — Baseline (upstream v0.7.13, no patches)

Metric	Value
Trace ID	`00000000aaaa11112222333344445555`
Observations	66
Services in trace	`platform_assistant` only
AgentGateway spans	0 ❌

Problem: Agent uses the default CompositeTextMapPropagator but never injects traceparent into outbound HTTP headers. LLM calls to agentgateway-proxy start a new disconnected trace.

Stage 1 — Commit 1 only (Python SDK: W3C propagator + AioHttpClientInstrumentor)

Sent request directly to agent pod (bypassing controller) to isolate the Python SDK changes.

Metric	Value
Trace ID	`11111111bbbb22223333444455556666`
Observations	130
Services in trace	`platform_assistant` + `agentgateway-proxy`
AgentGateway spans	4 ✅

Result: TraceContextTextMapPropagator + AioHttpClientInstrumentor causes the agent's outbound LLM calls (via litellm's aiohttp transport) to inject traceparent/tracestate headers. AgentGateway sees the trace context and its spans now join the same trace.

Stage 2 — Both commits (Python SDK + Go controller trace propagation)

A2A request sent through the controller (the production path).

Before (upstream controller, patched agent image)

Metric	Value
Trace ID sent	`33333333dddd44445555666677778888`
Trace found in Langfuse	NOT FOUND ❌
Orphan trace ID	`1159632b4b40815148c345bc9ead6dbf`
Orphan observations	64
Orphan services	`platform_assistant` + `agentgateway-proxy` + `agentgateway-waypoint`

Problem: The upstream controller strips traceparent when proxying A2A requests to agent pods. The agent creates a new root trace (1159632b...), disconnected from the caller's trace. The agent's outbound calls to agentgateway DO propagate correctly (thanks to Commit 1), but on the wrong trace ID.

After (patched controller + patched agent)

Metric	Value
Trace ID sent	`22222222cccc33334444555566667777`
Trace found in Langfuse	YES ✅
Observations	69
Services in trace	`platform_assistant` + `agentgateway-proxy`

Result: The patched controller extracts traceparent/tracestate from the incoming A2A request and re-injects them when proxying to the agent pod. The full chain — caller → controller → agent → agentgateway → LLM — now shares a single unified trace ID.

Summary

What	Before	After
Agent → AgentGateway propagation	❌ Disconnected	✅ Same trace
Controller → Agent propagation	❌ traceparent dropped	✅ traceparent forwarded
End-to-end trace unity	❌ 2-3 disconnected traces	✅ Single trace across all services

syn-zhu · 2026-02-26T04:22:46Z

syn-zhu · 2026-02-26T04:25:28Z

Hi @syn-zhu, Could you please test this locally on your end first? As it's Claude-generated code, a brief manual validation, e.g. such as posting before/after screenshots from a tracing tool would be the minimum step to ensure it's ready for contribution.

updated

krisztianfekete

This looks mostly good, but can you please look at the two comments I've just added?

krisztianfekete · 2026-02-26T14:50:32Z

python/packages/kagent-core/src/kagent/core/tracing/_utils.py

        logging.info("Enabling tracing")
+        # Set up W3C TraceContext propagator so incoming traceparent headers
+        # are extracted and outgoing requests carry them forward.
+        set_global_textmap(CompositeHTTPPropagator([TraceContextTextMapPropagator()]))


Are you sure this is necessary? If it is (but I don't think it is), we should at least preserve the existing propagators that this overrides.

Good catch, no it isn't necessary anymore. addressed in most recent commit

krisztianfekete · 2026-02-26T14:51:53Z

go/core/pkg/auth/auth.go

These are unrelated to auth. Can we move these into internal/ or somewhere tracing-specific?

addressed in most recent commits!

Two changes to enable end-to-end W3C TraceContext propagation: 1. Add AppProtocol "kgateway.dev/a2a" to agent Service port so AgentGateway can discover agent Services directly via kgateway protocol matching, rather than proxying through the controller. Update all golden test outputs to include the new appProtocol field. 2. Set up W3C TraceContext propagator in the Python agent SDK tracing configuration so agent pods correctly extract incoming traceparent headers and propagate them on outgoing requests. Fixes kagent-dev#1295 Signed-off-by: Simon Zhu <simon.zhu@mongodb.com>

…t pods The A2A server deserializes incoming HTTP requests into JSON-RPC params, discarding the original HTTP headers. When the controller forwards requests to agent pods via the A2A client, trace context headers (traceparent, tracestate) are lost, breaking distributed tracing. Fix: capture W3C trace context headers from the incoming request into the Go context in the A2A auth middleware, then inject them into outgoing requests in the A2ARequestHandler. This closes the gap between the A2A server (which strips headers) and the A2A client (which constructs new HTTP requests). Also update the agent_with_passthrough golden test (added in kagent-dev#1327) to include the appProtocol field. Signed-off-by: Simon Zhu <simon.zhu@mongodb.com>

- Remove unnecessary set_global_textmap override in Python tracing setup; the OTEL SDK already configures TraceContext + W3CBaggage propagators by default. - Move trace header context utilities (TraceHeadersFrom/To) from go/core/pkg/auth into go/core/internal/tracecontext, since they are unrelated to auth. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Simon Zhu <simon.zhu@mongodb.com>

onematchfox · 2026-03-05T10:16:29Z

go/core/internal/controller/translator/agent/adk_api_translator.go

+				Name:        "http",
+				Port:        dep.Port,
+				TargetPort:  intstr.FromInt(int(dep.Port)),
+				AppProtocol: ptr.To("kgateway.dev/a2a"),


Whilst I'm completely for integrating with other projects, should this really be a hard-coded value that is applied to all Agents (regardless of whether users actually make use of KGateway/AgentGateway)?

What happens when this value changes; there seems to be a bit of a split going on between kgateway and agentgateway - so what would happen if this changes from kgateway.dev/a2a to agentgateway.dev/a2a? And what happens if I want to use a different appProtocol for whatever reason? Or, what happens if I only want some agents (not all) to be picked up via the A2A plugin?

Maybe a better approach here would be to allow users to configure this via the Agent CRD instead?

Maybe a better approach here would be to allow users to configure this via the Agent CRD instead?

Completely agree, and was just thinking to make this change myself. Will update the PR

onematchfox · 2026-03-05T10:26:54Z

python/packages/kagent-core/src/kagent/core/tracing/_utils.py

            logging.info("Created new TracerProvider")

        HTTPXClientInstrumentor().instrument()
+        _instrument_aiohttp_client()


What value do we get out of this? Outbound LLM calls are already instrumented from what I can tell - see openai.chat span for example.

See the Stage 0 vs Stage 1 comparison in my earlier comment: #1297 (comment)

The problem is correlation with the incoming request; if the incoming request to the agent had trace headers, they currently don't get propagated. the outbound calls start a completely disjoint trace.

Adding this change makes it possible to trace long, complex agent request flows (e.g. user calls Agent 1 who calls Agent 2 and Agent 3, who each call MCP server etc etc) with a single root trace ID

Does this make sense?

Having said that, I actually wanna triple-check this earlier comment: a48fbcf#r2859471141

Need to make sure that change which I removed is really not needed. Might need to add it back.

…opagation (#1433) This PR adds OpenTelemetry distributed tracing to the kagent controller API, fixes trace context propagation across A2A agent calls, and cleans up noise in the existing Python agent tracing. Fixes #1295 essentially replacing a chunk of #1297 (it does not address being able to set the `appProtocol` on the `Agent` `Service` - that is a separate concern IMO). Signed-off-by: Brian Fox <878612+onematchfox@users.noreply.github.com>

Copilot AI review requested due to automatic review settings February 14, 2026 01:01

syn-zhu requested review from EItanya, ilackarms and yuval-k as code owners February 14, 2026 01:01

Copilot started reviewing on behalf of syn-zhu February 14, 2026 01:02 View session

Copilot AI reviewed Feb 14, 2026

View reviewed changes

go/core/internal/controller/translator/agent/adk_api_translator.go Show resolved Hide resolved

syn-zhu closed this Feb 20, 2026

syn-zhu reopened this Feb 24, 2026

syn-zhu force-pushed the fix/agent-service-a2a-appprotocol branch from 13799c1 to c2c5902 Compare February 24, 2026 11:12

syn-zhu requested a review from peterj as a code owner February 24, 2026 11:12

syn-zhu changed the title ~~fix: set appProtocol on agent Services for A2A discovery~~ fix: propagate trace context and enable A2A discovery for agent Services Feb 24, 2026

syn-zhu force-pushed the fix/agent-service-a2a-appprotocol branch from c2c5902 to 256bbda Compare February 24, 2026 11:16

syn-zhu changed the title ~~fix: propagate trace context and enable A2A discovery for agent Services~~ fix: propagate trace context end-to-end for agent Services Feb 24, 2026

syn-zhu force-pushed the fix/agent-service-a2a-appprotocol branch from ffe2d68 to f965911 Compare February 24, 2026 11:28

syn-zhu force-pushed the fix/agent-service-a2a-appprotocol branch from f965911 to 3d2c566 Compare February 26, 2026 04:21

syn-zhu force-pushed the fix/agent-service-a2a-appprotocol branch from 3d2c566 to 32027e9 Compare February 26, 2026 04:25

krisztianfekete reviewed Feb 26, 2026

View reviewed changes

syn-zhu force-pushed the fix/agent-service-a2a-appprotocol branch 4 times, most recently from ae82ed1 to c234441 Compare March 5, 2026 09:26

syn-zhu added 2 commits March 5, 2026 04:27

syn-zhu force-pushed the fix/agent-service-a2a-appprotocol branch from c234441 to 5fd64e0 Compare March 5, 2026 09:27

syn-zhu requested a review from krisztianfekete March 5, 2026 09:28

onematchfox mentioned this pull request Mar 5, 2026

feat: instrument controller with distributed tracing and A2A trace propagation #1433

Merged

onematchfox reviewed Mar 5, 2026

View reviewed changes

Conversation

syn-zhu commented Feb 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

EItanya commented Feb 16, 2026

Uh oh!

syn-zhu commented Feb 20, 2026

Uh oh!

syn-zhu commented Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

krisztianfekete commented Feb 24, 2026

Uh oh!

syn-zhu commented Feb 26, 2026

End-to-End Test Results: Trace Propagation

Setup

Stage 0 — Baseline (upstream v0.7.13, no patches)

Stage 1 — Commit 1 only (Python SDK: W3C propagator + AioHttpClientInstrumentor)

Stage 2 — Both commits (Python SDK + Go controller trace propagation)

Before (upstream controller, patched agent image)

After (patched controller + patched agent)

Summary

Uh oh!

syn-zhu commented Feb 26, 2026

Uh oh!

syn-zhu commented Feb 26, 2026

Uh oh!

krisztianfekete left a comment

Choose a reason for hiding this comment

Uh oh!

krisztianfekete Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

syn-zhu Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

krisztianfekete Feb 26, 2026

Choose a reason for hiding this comment

Uh oh!

syn-zhu Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

onematchfox Mar 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

syn-zhu Mar 7, 2026

Choose a reason for hiding this comment

Uh oh!

onematchfox Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

syn-zhu Mar 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

syn-zhu Mar 7, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

syn-zhu commented Feb 14, 2026 •

edited

Loading

syn-zhu commented Feb 24, 2026 •

edited

Loading

krisztianfekete Feb 26, 2026 •

edited

Loading

onematchfox Mar 5, 2026 •

edited

Loading

syn-zhu Mar 7, 2026 •

edited

Loading