Idea: POC classic histograms emulated over NHCB

[krajorama]

What is the problem you are trying to solve?

Related to prometheus/prometheus#16948

Goal

Make a prototype implementation of emulating classic histograms in PromQL with NHCB to find out the overhead in terms of development and resources.

Input documentation

Send native histograms with custom buckets to Grafana Mimir.
Prometheus native histograms.

Requirements

• Per-tenant configurable experimental feature.
• No configuration beyond enable/disable for the end-user.
• No change required from user on PromQL queries.
• May have configuration for operator (limits/cache/etc).
• When enabled, if a user asks for an <metric>_bucket, <metric>_count or <metric>_sum based instance vector selector in PromQL instant/range query, then return samples that match the selector and also synthesize the requested samples from NHCB series matching the <metric> based instance vector selector. If there's a conflict, the classic series with the suffix wins.
• For range vector selector, the same logic applies as above and the decision to what sample needs to be included has to be decided on a per time step bases. This is due to having to correctly work during transient states:
  • NHCB starting to be scraped along classic histograms
  • NHCB is starting to be scrape instead of classic histograms
  • classic histograms are no longer scraped, only NHCB
• Optimize first for latency and resource utilization and simplicity next.

Test plan

What we want to measure is the overhead of doing this emulation over rewriting queries for native histograms. This helps decide what strategy to follow when migrating a user from classic histograms to native histograms with custom buckets and indeed to make this feature GA.

Ideally we'd use two Mimir installations with the same base load. Then run the classic queries emulated over NHCB on one instance and at the same time run the migrated NHCB query on the other instance. Then compare the latency and resource utilization.

However we don't want to implement a straight rewrite of queries to native histograms queries, because a straight rewrite would not handle the transient states (see the warning in the Grafana Mimir docs).

Since at this stage we're only changing the implementation of instant and range vector selectors, a synthetic test targeting these should be enough. Measure these against each other, both instant and range query:

• select <metric>_count vs histogram_count(<metric>)
• select <metric>_sum vs histogram_sum(<metric>)
• select <metric>_bucket vs <metric> (retrieve all buckets for a heatmap for example)
• select <metric>_bucket{le="<x>"} vs (histogram_fraction(-Inf, <x>, <metric>) < +Inf) * histogram_count(<metric>) vs histogram_count(<metric>) </ <x> (SLO query, trim operator test)

Measure on data that has a time range without NHCB, time range with both NHCB and classic, and NHCB only.
Test ranges should overlap with steady and 50-50% transient states as well.

These above can be used for correctness tests as well.

Next possible steps

Out of scope for now, these seem hard to implement with the exception of count, sum series.

• Emulating <metric>_count or <metric>_sum from exponential histograms.
• Emulating histogram_quantile(<x>, sum by (le) <metric>_bucket) from exponential histograms.
• Emulating <metric>_bucket{le="<x>",...} from exponential histograms (emulate with trim operator).

Out of scope

• Matching series with regex matcher on ___name__.
• Non PromQL endpoints beyond instant and range query.

Which solution do you envision (roughly)?

TBD to explore.

Use existing or new cache to avoid overhead for metrics where we don't have NHCB present.

Have you considered any alternatives?

Yes. If overhead is too high, we will not prioritize this.

Any additional context to share?

No response

How long do you think this would take to be developed?

Medium (~2 months dev)

What are the documentation dependencies?

No response

Proposer?

No response