Skip to content

[BUG] Panic converting OTLP exponential histograms with low scale #55140

Description

@gillesbergerp

Agent version

opentelemetry-mapping-go v0.29.0

Bug Report

A producer that downscales to its floor takes the whole collector down — the panic lands in an exporter-queue goroutine, so nothing upstream can recover it. scale down to -10 is spec-valid, so this is well-formed input.

gamma in exponentialHistogramToDDSketch is computed but never bounded:

gamma := math.Pow(2, math.Pow(2, float64(-p.Scale())))

histograms.go already clamps the identical line so the same guard on the OTLP path should be enough:

gammaWithOnePercentAccuracy := 1.01 / 0.99
gamma := math.Pow(2, math.Pow(2, float64(-scale)))
gamma = math.Min(gamma, gammaWithOnePercentAccuracy)

Reproduction Steps

Observed datapoint

-> Name: test.historgram
-> Unit: s
-> DataType: ExponentialHistogram
-> AggregationTemporality: Delta
Count: 3   Sum: 0.627158   Min: 0.143741   Max: 0.327218

161 buckets, reconstructed boundaries:
   85x  (0.000000, 0.000000]
   72x  (+Inf, +Inf]
    1x  [-1.34e152, -1.000000)
    1x  (0.000000, 1.000000]
    1x  (1.000000, 1.34e152]
    1x  (1.34e152, +Inf]

Stack trace

panic: runtime error: slice bounds out of range [-4611686018427387209:]

goroutine 275 [running]:
sketches-go@v1.4.7/ddsketch/store.(*DenseStore).shiftCounts        dense_store.go:115
sketches-go@v1.4.7/ddsketch/store.(*DenseStore).centerCounts(0xc00217d300, 0x8000000000000000, 0x5ea4)
                                                                   dense_store.go:107
sketches-go@v1.4.7/ddsketch/store.(*DenseStore).extendRange(0xc00217d300, 0x7ff0000000000000?, 0x0?)
                                                                   dense_store.go:95
sketches-go@v1.4.7/ddsketch/store.(*DenseStore).AddWithCount(0xc00217d300, 0x8000000000000000, 0xfff8000000000000)
                                                                   dense_store.go:55
sketches-go@v1.4.7/ddsketch.(*DDSketch).ChangeMapping              ddsketch.go:486
opentelemetry-mapping-go/pkg/quantile.createDDSketchWithSketchMapping   ddsketch.go:52
opentelemetry-mapping-go/pkg/quantile.ConvertDDSketchIntoSketch         ddsketch.go:200
opentelemetry-mapping-go/pkg/otlp/metrics.(*Translator).mapExponentialHistogramMetrics
                                                exponential_histograms_translator.go:148
opentelemetry-mapping-go/pkg/otlp/metrics.(*Translator).mapToDDFormat   metrics_translator.go:888
opentelemetry-mapping-go/pkg/otlp/metrics.(*Translator).MapMetrics      metrics_translator.go:827
datadog-agent/.../serializerexporter.(*Exporter).ConsumeMetrics         exporter.go:192
[... exporterhelper frames ...]
created by ...queue.(*asyncQueue[...]).Start in goroutine 1  async_queue.go:39

Agent configuration

  • otelcol: v0.129.0
  • go.opentelemetry.io/collector/exporter: v0.129.0
  • go.opentelemetry.io/collector/consumer: v1.35.0
  • DataDog/opentelemetry-mapping-go/pkg/otlp/metrics: v0.29.0
  • DataDog/opentelemetry-mapping-go/pkg/quantile: v0.29.0
  • DataDog/sketches-go: v1.4.7
  • DataDog/datadog-agent/.../serializerexporter: v0.68.0-devel.0.20250604170009-feaecfbfd700

Operating System

No response

Other environment details

No response

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions