Distributed tracing context in ADK

[usuallyunusual]

Hello!
I'm trying out ADK and I wanted to know if we can 'export' trace context from ADK to downstream HTTP calls.
Description :
We have an MCP server running (StreamableHTTP) and I wanted to the ADK deployment on AgentEngine to connect to this via the MCPToolset along with traces. The connection is working fine, the agent recognizes the tools and instruments all of the calls as well but only within it's server boundary. So once it calls one of the MCP Tools on another server, it essentially black-boxes it in the trace. Our MCP tools are potentially be multistep processes that require their own spans. Now I can implement my own instrumentation and tracing on our MCP server but I would like to somehow be able to pass the existing trace-context from the ADK created root-span and pass it along to the MCP server to import, so our traces maintain coherence.
I checked out the repo and could not find anything that solves my use-case here. Please let me know if something like this can be implemented.
Open toe contribute into the project if required.
Thanks!

[adk-bot]

Response from ADK Answering Agent (experimental, answer may be inaccurate)

TLDR: Yes, this is possible! ADK uses the OpenTelemetry standard for tracing. To connect the traces, you don't need to manually export context from ADK. Instead, you need to instrument your MCP server with an OpenTelemetry SDK, which will automatically pick up the trace context ADK sends in the HTTP request headers.

---

Hello! That's an excellent question, and you've hit on a core concept of modern application observability. What you're looking to achieve is called "distributed tracing," and the good news is that the ADK ecosystem is designed to support this.

The key is that ADK's tracing capabilities are built on the OpenTelemetry (OTEL) standard. [1] Here’s how it works for your use case:

1. Automatic Context Propagation: When your ADK agent is running and a trace is initiated, ADK's internal OpenTelemetry tracer creates and maintains a trace context (which includes a trace_id and span_id).
2. HTTP Header Injection: When your agent calls an external service via a tool (like your MCPToolset making an HTTP call), the OpenTelemetry SDK automatically injects the current trace context into the HTTP headers of that outgoing request. This is typically done using the standard W3C traceparent and tracestate headers.
3. Receiving Context on Your Server: For your MCP server to continue the trace, it also needs to be instrumented with an OpenTelemetry SDK. Your server-side OTEL SDK will automatically inspect incoming HTTP requests for the traceparent header. When it finds one, it extracts the context and starts its own spans as children of the span from the calling ADK agent.

This is what links the two systems together into a single, coherent trace, removing the "black box" you're seeing.

What you need to do

You don't need to change anything on the ADK side. Your focus should be on your MCP server. You need to integrate an OpenTelemetry SDK into it. The specific implementation will depend on the language and framework your MCP server is built with, but the general steps are:

1. Add the appropriate OpenTelemetry SDK libraries to your MCP server project.
2. Configure an exporter to send the traces to your observability backend of choice (like Google Cloud Trace, AgentOps, Arize, etc., which are all supported by ADK). [1, 2, 3]
3. Ensure the server framework instrumentation is active, so it automatically handles the extraction of context from incoming HTTP headers.

The official OpenTelemetry documentation is the best resource for how to instrument your specific service. The documentation for mcp-toolbox-for-databases also mentions OpenTelemetry support, which confirms this is an intended pattern in the MCP ecosystem. [4]

Thank you for your willingness to contribute! In this case, since the mechanism relies on instrumenting your own service, no contribution to the ADK repository is needed.

---

[1] https://cloud.google.com/trace/docs/overview
[2] https://www.agentops.ai
[3] https://docs.arize.com/arize
[4] https://github.com/modelcontextprotocol/mcp-toolbox-for-databases

[usuallyunusual]

This does not seem to work. Although my MCP server is able to see the trace and extract it, pushing to CloudTrace is not working.
I have tried running the service locally and pushing logs to cloud-trace, which is working fine with existing traceparent and without as well (In which case, it creates fresh trace). So I know my setup is working fine. Once I deploy to cloud-run it stops working even though the logs on my cloud-run indicate it's picking up the traces correctly but they do not show up on CloudTrace.
I see other spans from my cloud-run being collected by cloudtrace in the same span. But anything that's a child-span of the execute_tool xyz does not seem to be collected by cloud-trace

[usuallyunusual]

Adding more info for this.

The trace context is not propagated automatically. I had to do it manually.
I am manually injecting trace-headers into the outgoing request in ADK through the header_provider for the MCPToolset using a StreamableHTTPServerParams.

 connection_params = StreamableHTTPServerParams(
            url=mcp_server_url,
            headers=None,  # Headers will be provided dynamically via header_provider
)
mcp_toolset = MCPToolset(
    connection_params=connection_params,
    header_provider=get_mcp_headers,
)

My supplier basically get's current active span and pushes it through to downstream Cloud Run MCP server. The traceID and spanID match what's shown on cloud-trace for execute_tool xyz span.

On my MCP CloudRun service (FastAPI + gunicorn + CloudRun Service). we had issues like spanID being changed when the request came to us. This spanID was not existing in cloud-trace and some sources on the internet say the infra layer before my service adds it's own spans.
So I decided to create my own trace-header. Along with the usual traceparent I also send in a x-preserved-traceparent which I actually read downstream. This worked perfectly and I am able to align AgentEngine service and cloudrun.
Something like,

class CustomTraceContextPropagator(TraceContextTextMapPropagator):
    """
    Custom trace context propagator that reads from x-preserve-traceparent header
    instead of the default traceparent header.
    """
    _TRACEPARENT_HEADER_NAME = "x-preserve-traceparent"

My textmap propagators look like this

 set_global_textmap(
            CompositePropagator([CustomTraceContextPropagator(),TraceContextTextMapPropagator(), CloudTraceFormatPropagator()])
        )

I have also added a gunicorn shutdown hook to ensure spans are flushed. Also added an atexit hook to flush spans.
I've also disabled CPU throttling --no-cpu-throttling

Finally I have also tried nuking every span attribute, events and links

class AttributeNuker(SpanProcessor):
    def on_end(self, span):
        if hasattr(span, "_attributes"):
             span._attributes = {}
        
        if hasattr(span, "_events"):
            span._events = []

        if hasattr(span, "_links"):
            span._links = []

From local service runs everything runs perfectly fine (Importing from existing trace and new traces) and everything is pushed to cloud-trace perfectly.

From cloudrun some of my spans are coming through i.e MCP tools/list calls are coming through but any span that I add to execute_tool xyz span just does not seem to come through.

Enabling debug logging tells me

https://telemetry.googleapis.com:443 "POST /v1/traces HTTP/1.1" 200 0

so I'm assuming export is happening.

Any help here is appreciated.

Metadata

google-cloud-aiplatform==1.128.0
google-adk==1.19.0