Tail sampling exports spans that end after the root of a dropped trace

[petyosi]

Summary

TailSamplingProcessor releases a trace's buffer as soon as the root span ends. Any span of that trace that ends later finds no buffer, takes the pass-through path, and is exported unconditionally — including for traces the sampler explicitly dropped.

The same release has a second consequence: once the root has ended, a span that does meet the sampling criteria can no longer include its trace, so the trace arrives at the backend as a rootless fragment.

I found this while reviewing the JavaScript port of this processor. Filing here first, because the JS SDK is a faithful port and a fix there would diverge from this implementation — see pydantic/logfire-js#229 for the JS report and pydantic/logfire-js#230 for a proposed JS fix. We would rather settle the shape upstream than let the two SDKs drift.

Where

logfire/sampling/_tail_sampling.py, in on_end:

if span.parent is None:
    # This is the root span, so the trace is hopefully complete.
    # Delete the buffer to save memory.
    self.traces.pop(trace_id, None)

and then, below the lock:

if buffer is None:
    # No buffer for this trace, meaning it was already sampled/discarded, or never tracked.
    # Pass on_end through immediately to both processors.
    super().on_end(span)
    if self.deferred_processor is not None:
        self.deferred_processor.on_end(span)

The comment enumerates "already sampled/discarded, or never tracked". After a root ends it also catches spans belonging to traces that were dropped, and those are passed through as though they had never been tracked.

The root ending is not a reliable signal that a trace is complete. Anything detached from the request that outlives it — a background task, a fire-and-forget coroutine, a queue publish, a streaming response finalizer — ends after the root.

Repro 1: a dropped trace still exports a span

import logfire
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter

exporter = InMemorySpanExporter()

logfire.configure(
    send_to_logfire=False,
    console=False,
    additional_span_processors=[SimpleSpanProcessor(exporter)],
    # Never sample anything: nothing should ever be exported.
    sampling=logfire.SamplingOptions(tail=lambda info: 0.0),
)

root = logfire.span('root')
root.__enter__()
child = logfire.span('child')
child.__enter__()
# The root finishes while the child is still running.
root.__exit__(None, None, None)
child.__exit__(None, None, None)

print([s.name for s in exporter.get_finished_spans()])

Expected [], actual:

['child']

Repro 2: an error after the root ends produces a rootless trace

import logfire
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter

exporter = InMemorySpanExporter()

logfire.configure(
    send_to_logfire=False,
    console=False,
    additional_span_processors=[SimpleSpanProcessor(exporter)],
    sampling=logfire.SamplingOptions.level_or_duration(duration_threshold=None),
)

root = logfire.span('root')
root.__enter__()
child = logfire.span('child')
child.__enter__()
root.__exit__(None, None, None)
# The child fails, but only after the root has already ended.
child.set_level('error')
child.__exit__(None, None, None)

spans = exporter.get_finished_spans()
print([s.name for s in spans], 'trace ids:', len({s.context.trace_id for s in spans}))

Expected the whole trace (root and child), actual:

['child'] trace ids: 1

check_span is never reached for the late span, because it only runs while a buffer exists. So the error neither includes its trace nor is dropped — it is exported alone, and the trace shows up in Logfire with the failing span and no root.

Of the two, this second one looks like the more damaging in practice: tail sampling by level exists precisely to catch errors, and an error raised in work that outlives the request produces a broken trace instead of a complete one.

Notes towards a fix

A plain tombstone marking "this trace was dropped" is the obvious approach but never gets cleaned up: today the buffer is only removed when the root ends, and by tombstone time the root has already ended, so nothing would ever clear it.

Tying the lifetime to outstanding spans is bounded without needing an eviction policy: count started-but-not-yet-ended spans, and release the trace only once the root has ended and that count reaches zero. Root end then stops being a special case and is simply the last end in the ordinary ordering, leaving the common path unchanged. That is the shape implemented in pydantic/logfire-js#230, which also caps how many traces may be held past their root so that a span which never ends cannot pin its trace forever.

The tradeoff is memory: a trace with a leaked or very long-lived span keeps its buffer, where today it is released promptly. That is the delicate part of this processor, so I would rather have your call on the shape than send a patch straight away. Happy to open a PR here once you have a preference, or just the failing tests if you would rather implement it yourself.

Environment

• logfire 4.36.0
• opentelemetry-sdk 1.42.1
• Python 3.14.3