[Tracing plugin] Add a `mergeItems` option akin to that of `@opentelemetry/instrumentation-graphql`

[acidoxee]

The default GraphQL instrumentation from @opentelemetry/instrumentation-graphql supports a mergeItems option. It enables grouping all list items under a single span (like users.*.name) instead of having one span per list item (users.0.name, users.1.name, etc.).

Would it be possible to have such an option in @pothos/plugin-tracing? Either as a top-level, configure-once-for-all option (like the default instrumentation's option mentioned above), or even better, as a per-resolver option (which could enable business-related decisions for grouping, such as keeping unimportant list resolvers grouped, but splitting important ones where being able to examine each list item's resolver individually makes sense)?

Right now, I believe having each list item's resolver distinct from one another is the best default, but since large lists can generate quite a lot of spans, it would be nice to have the option to tone it down a notch. Managed OTel backends such as Sentry charge by the amount of spans generated, so this could be a nice way to reduce spans in non-critical spots.

[hayes]

yes, I think this seems reasonable. There isn't currently a good way to do this that I can think of, but we can definitely add something that would allow this.

The mergeItems option is somewhat misleading because its really just capturing the first item and ignoring everything else. Are you after real merging/grouping, or just filtering?

Adding a way to filter spans based on path would be the easiest: something like: filter: (path) => !isNotFirstItem(path) would be both easy and flexible.

Maybe something more useful would be an aggregate span that has:

• start: start of first span
• end: end of last span
• count
• average duration
• min duration

This would be harder to implement, but would allow you to get more data without extra spans

There is probably also a question about how nested lists should be handled. I think generally a solution that could allow you to group all child fields into a parent span could be interesting as well

[acidoxee]

I'm not fond of the filtering approach, even if that could quickly solve the issue regarding the amount of spans. I'm not sure where child spans (all those generated from what the resolvers call) would be attached. A single aggregate span, as you suggest, was more my intention.

I'm not entirely sure which OTel primitives could be leveraged to still have enough information for debugging that single span, but span events could be one of them for instance. This is obviously a degraded signal compared to proper spans, which are the best signal for duration-based tracking, but it might be "the next best thing". Tradeoffs!

[hayes]

I did some initial investigation into trying to merge traces, but didn't come up with anything I loved. If you can provide a more technical description of what you would actually want that could be helpful. Doing what the open telemetry instrumentation does would be easy, but sounds like its not what you are looking for.

I couldn't find good examples of aggregate spans, and the logic to merge/combine them automatically gets pretty weird and complicated when you get into multiple levels of nested lists. Because of the async nature of requests one path in your query can have multiple spans that do not overlap. This makes it really hard to know when to finalize/end a span for a list item.

I am very open to suggestions on how to approach this, but currently this is a little too open ended (and I don't have a good test case to build for)

[acidoxee]

Thanks for taking the time to investigate. I may have misunderstood what the GraphQL OTel instrumentation is doing, but here, they state:

Notice that all span data only reflects the invocation on the first element. That includes timing, events and status.

Downstream spans in the context of all resolvers will be child of the first span.

The second sentence is more what I was worried about, which is that all child spans of the list should still be attached to that single parent span. If you're able to replicate this, it could already be a win.

The first sentence is indeed where solving it would get technical. As I understand it, that span will indeed not be an accurate "aggregate" of all child spans, but just reflect the first child's, which could prove inaccurate if other children take longer or are in a failed state for instance. Ideally, I'd like that single aggregate span to have the longest timing of all (earliest start, latest finish), the worst status (if any child resolver fails, but not the first one, the single span should still be marked as failed), and so on. I understand though if that's too much of a hassle 👍

[hayes]

To clarify what the open tracing plugin does:

Given a query like:

query {
  search(query: "a") {
    __typename
    id #assume search return a node interface with id, and User and Post both implement Node
    ...on User {
       name
       profile: { bio }
    }
    ... on Post {
      title
   }
  }
}

and a result like:

{
  search: [
    { __typename: 'User', name: "a", { profile: null } },
    { __typename: 'Post', title: "a" },
    { __typename: 'User', name: `"b" { profile: { bio: "I'm b" }},
    { __typename: 'Post', name: "b" },
  ]
}

open telemetry with mergeItems would return something like:

• search
• search.*.id (timing from search[0].id)
• search.*.name (timing from search[0].name)
• search.*.title (timing from search[1].title
• search.*.profile(timing from search[0].profile)
• search.*.profile.bio (timing from search[2].profile.bio

This is still a very simplistic example, but it shows how this can get complicated fairly quickly, and what mergeItems does is pretty basic.

It literally just captures the first timing result for any given path with indexes normalized to *. There are a bunch of things that complicate this and make the results unintuitive:

• Polymorphic types (unions and interfaces) have fields with the same name but different resolvers and get merged
• "first" doesn't mean first in the list, it means first resolved and can change depending on null values, timing, where in a tree its being resolved, etc
• most data is just thrown away

Pothos mostly just works by wrapping individual resolvers, or by adding metadata to field extensions. It isn't well suited to doing aggregation across different parts of a query because it doesn't really have any insight into the broader query execution flow.

For things that need broader execution lifecycle hooks envelop plugins for graphql-yoga can be a good option. I am definitely open to figuring out how to do this better in pothos, but I don't see an obvious way tog actually implement it yet

[acidoxee]

I see, this is indeed not what I thought it was doing. I had a simpler thing in mind, where the first resolver would entirely dictate the rest of the tree under it. But as you mentioned, the more complex aggregations don't seem easy to reproduce, and I also didn't take interfaces into account.

No worries then. If you ever think of some solution, I'll be happy to try it out on our schema, but in any case we'll have to work on sampling sooner or later.