feat(llmobs): success criteria assessment for custom evals #14792

Yun-Kim · 2025-10-07T18:26:08Z

Description

Adds success_asssesment as an argument to submit_evaluation_for(). This arg is used to denote whether or not the submitted evaluation is correct/valid (particularly in the context of evaluations using LLM-as-a-judge). Provided values must be "pass" or "fail".

Testing

Risks

Additional Notes

github-actions · 2025-10-07T18:26:50Z

CODEOWNERS have been resolved as:

releasenotes/notes/feat-llmobs-submit-eval-success-assessment-11b93921832c54b4.yaml  @DataDog/apm-python
ddtrace/llmobs/_llmobs.py                                               @DataDog/ml-observability
ddtrace/llmobs/_writer.py                                               @DataDog/ml-observability
tests/llmobs/_utils.py                                                  @DataDog/ml-observability
tests/llmobs/test_llmobs_service.py                                     @DataDog/ml-observability

github-actions · 2025-10-07T18:51:41Z

Bootstrap import analysis

Comparison of import times between this PR and base.

Summary

The average import time from this PR is: 242 ± 3 ms.

The average import time from base is: 246 ± 4 ms.

The import time difference between this PR and base is: -4.0 ± 0.2 ms.

Import time breakdown

The following import paths have shrunk:

ddtrace.auto 2.152 ms (0.89%)

ddtrace.bootstrap.sitecustomize 1.472 ms (0.61%)

ddtrace.bootstrap.preload 1.472 ms (0.61%)

ddtrace.internal.remoteconfig.client 0.700 ms (0.29%)

ddtrace 0.680 ms (0.28%)

ddtrace.internal._unpatched 0.030 ms (0.01%)

json 0.030 ms (0.01%)

json.decoder 0.030 ms (0.01%)

re 0.030 ms (0.01%)

enum 0.030 ms (0.01%)

types 0.030 ms (0.01%)

pr-commenter · 2025-10-07T19:22:56Z

Performance SLOs

Comparing candidate yunkim/evals-success-criteria (b9840bb) with baseline main (8235d03)

📈 Performance Regressions (3 suites)

📈 iastaspects - 118/118

✅ add_aspect

Time: ✅ 0.402µs (SLO: <10.000µs 📉 -96.0%) vs baseline: -0.2%

Memory: ✅ 38.044MB (SLO: <39.000MB -2.5%) vs baseline: +4.7%

✅ add_inplace_aspect

Time: ✅ 0.412µs (SLO: <10.000µs 📉 -95.9%) vs baseline: +1.4%

Memory: ✅ 38.044MB (SLO: <39.000MB -2.5%) vs baseline: +4.7%

✅ add_inplace_noaspect

Time: ✅ 0.317µs (SLO: <10.000µs 📉 -96.8%) vs baseline: -0.4%

Memory: ✅ 38.083MB (SLO: <39.000MB -2.4%) vs baseline: +4.8%

✅ add_noaspect

Time: ✅ 0.279µs (SLO: <10.000µs 📉 -97.2%) vs baseline: ~same

Memory: ✅ 38.063MB (SLO: <39.000MB -2.4%) vs baseline: +5.1%

✅ bytearray_aspect

Time: ✅ 1.315µs (SLO: <10.000µs 📉 -86.9%) vs baseline: ~same

Memory: ✅ 38.103MB (SLO: <39.000MB -2.3%) vs baseline: +5.0%

✅ bytearray_extend_aspect

Time: ✅ 1.458µs (SLO: <10.000µs 📉 -85.4%) vs baseline: +0.9%

Memory: ✅ 38.063MB (SLO: <39.000MB -2.4%) vs baseline: +4.8%

✅ bytearray_extend_noaspect

Time: ✅ 0.607µs (SLO: <10.000µs 📉 -93.9%) vs baseline: -1.4%

Memory: ✅ 37.965MB (SLO: <39.000MB -2.7%) vs baseline: +4.5%

✅ bytearray_noaspect

Time: ✅ 0.478µs (SLO: <10.000µs 📉 -95.2%) vs baseline: +0.4%

Memory: ✅ 38.044MB (SLO: <39.000MB -2.5%) vs baseline: +4.7%

✅ bytes_aspect

Time: ✅ 1.292µs (SLO: <10.000µs 📉 -87.1%) vs baseline: -0.2%

Memory: ✅ 38.083MB (SLO: <39.000MB -2.4%) vs baseline: +4.9%

✅ bytes_noaspect

Time: ✅ 0.492µs (SLO: <10.000µs 📉 -95.1%) vs baseline: -0.6%

Memory: ✅ 38.103MB (SLO: <39.000MB -2.3%) vs baseline: +5.0%

✅ bytesio_aspect

Time: ✅ 1.350µs (SLO: <10.000µs 📉 -86.5%) vs baseline: -0.7%

Memory: ✅ 38.063MB (SLO: <39.000MB -2.4%) vs baseline: +4.9%

✅ bytesio_noaspect

Time: ✅ 0.505µs (SLO: <10.000µs 📉 -95.0%) vs baseline: +1.4%

Memory: ✅ 38.122MB (SLO: <39.000MB -2.3%) vs baseline: +5.2%

✅ capitalize_aspect

Time: ✅ 0.738µs (SLO: <10.000µs 📉 -92.6%) vs baseline: +0.4%

Memory: ✅ 38.083MB (SLO: <39.000MB -2.4%) vs baseline: +4.7%

✅ capitalize_noaspect

Time: ✅ 0.435µs (SLO: <10.000µs 📉 -95.6%) vs baseline: -0.9%

Memory: ✅ 38.083MB (SLO: <39.000MB -2.4%) vs baseline: +4.9%

✅ casefold_aspect

Time: ✅ 0.736µs (SLO: <10.000µs 📉 -92.6%) vs baseline: -0.9%

Memory: ✅ 38.044MB (SLO: <39.000MB -2.5%) vs baseline: +4.9%

✅ casefold_noaspect

Time: ✅ 0.368µs (SLO: <10.000µs 📉 -96.3%) vs baseline: -0.4%

Memory: ✅ 38.103MB (SLO: <39.000MB -2.3%) vs baseline: +4.9%

✅ decode_aspect

Time: ✅ 0.723µs (SLO: <10.000µs 📉 -92.8%) vs baseline: ~same

Memory: ✅ 38.103MB (SLO: <39.000MB -2.3%) vs baseline: +5.0%

✅ decode_noaspect

Time: ✅ 0.416µs (SLO: <10.000µs 📉 -95.8%) vs baseline: -0.3%

Memory: ✅ 38.083MB (SLO: <39.000MB -2.4%) vs baseline: +5.0%

✅ encode_aspect

Time: ✅ 0.714µs (SLO: <10.000µs 📉 -92.9%) vs baseline: +1.4%

Memory: ✅ 38.083MB (SLO: <39.000MB -2.4%) vs baseline: +4.9%

✅ encode_noaspect

Time: ✅ 0.404µs (SLO: <10.000µs 📉 -96.0%) vs baseline: +1.1%

Memory: ✅ 38.083MB (SLO: <39.000MB -2.4%) vs baseline: +4.8%

✅ format_aspect

Time: ✅ 3.383µs (SLO: <10.000µs 📉 -66.2%) vs baseline: +0.1%

Memory: ✅ 38.024MB (SLO: <39.000MB -2.5%) vs baseline: +4.8%

✅ format_map_aspect

Time: ✅ 3.572µs (SLO: <10.000µs 📉 -64.3%) vs baseline: -1.1%

Memory: ✅ 38.044MB (SLO: <39.000MB -2.5%) vs baseline: +4.9%

✅ format_map_noaspect

Time: ✅ 0.780µs (SLO: <10.000µs 📉 -92.2%) vs baseline: +1.1%

Memory: ✅ 38.083MB (SLO: <39.000MB -2.4%) vs baseline: +4.9%

✅ format_noaspect

Time: ✅ 0.597µs (SLO: <10.000µs 📉 -94.0%) vs baseline: +0.5%

Memory: ✅ 38.083MB (SLO: <39.000MB -2.4%) vs baseline: +4.9%

✅ index_aspect

Time: ✅ 0.355µs (SLO: <10.000µs 📉 -96.4%) vs baseline: -0.9%

Memory: ✅ 38.103MB (SLO: <39.000MB -2.3%) vs baseline: +5.0%

✅ index_noaspect

Time: ✅ 0.279µs (SLO: <10.000µs 📉 -97.2%) vs baseline: -0.2%

Memory: ✅ 38.044MB (SLO: <39.000MB -2.5%) vs baseline: +4.9%

✅ join_aspect

Time: ✅ 1.397µs (SLO: <10.000µs 📉 -86.0%) vs baseline: +1.3%

Memory: ✅ 38.004MB (SLO: <39.000MB -2.6%) vs baseline: +4.6%

✅ join_noaspect

Time: ✅ 0.488µs (SLO: <10.000µs 📉 -95.1%) vs baseline: -0.8%

Memory: ✅ 38.044MB (SLO: <39.000MB -2.5%) vs baseline: +5.0%

✅ ljust_aspect

Time: ✅ 2.584µs (SLO: <20.000µs 📉 -87.1%) vs baseline: +4.7%

Memory: ✅ 38.063MB (SLO: <39.000MB -2.4%) vs baseline: +4.9%

✅ ljust_noaspect

Time: ✅ 0.405µs (SLO: <10.000µs 📉 -96.0%) vs baseline: +0.6%

Memory: ✅ 38.103MB (SLO: <39.000MB -2.3%) vs baseline: +5.1%

✅ lower_aspect

Time: ✅ 2.186µs (SLO: <10.000µs 📉 -78.1%) vs baseline: -1.1%

Memory: ✅ 38.122MB (SLO: <39.000MB -2.3%) vs baseline: +4.9%

✅ lower_noaspect

Time: ✅ 0.366µs (SLO: <10.000µs 📉 -96.3%) vs baseline: -0.2%

Memory: ✅ 38.103MB (SLO: <39.000MB -2.3%) vs baseline: +5.0%

✅ lstrip_aspect

Time: ✅ 2.606µs (SLO: <20.000µs 📉 -87.0%) vs baseline: 📈 +17.2%

Memory: ✅ 38.122MB (SLO: <39.000MB -2.3%) vs baseline: +4.9%

✅ lstrip_noaspect

Time: ✅ 0.384µs (SLO: <10.000µs 📉 -96.2%) vs baseline: +1.3%

Memory: ✅ 38.024MB (SLO: <39.000MB -2.5%) vs baseline: +4.6%

✅ modulo_aspect

Time: ✅ 0.995µs (SLO: <10.000µs 📉 -90.1%) vs baseline: -0.8%

Memory: ✅ 38.083MB (SLO: <39.000MB -2.4%) vs baseline: +4.8%

✅ modulo_aspect_for_bytearray_bytearray

Time: ✅ 1.538µs (SLO: <10.000µs 📉 -84.6%) vs baseline: +0.5%

Memory: ✅ 38.044MB (SLO: <39.000MB -2.5%) vs baseline: +4.7%

✅ modulo_aspect_for_bytes

Time: ✅ 0.981µs (SLO: <10.000µs 📉 -90.2%) vs baseline: +0.8%

Memory: ✅ 38.044MB (SLO: <39.000MB -2.5%) vs baseline: +4.7%

✅ modulo_aspect_for_bytes_bytearray

Time: ✅ 1.199µs (SLO: <10.000µs 📉 -88.0%) vs baseline: ~same

Memory: ✅ 38.103MB (SLO: <39.000MB -2.3%) vs baseline: +5.0%

✅ modulo_noaspect

Time: ✅ 0.631µs (SLO: <10.000µs 📉 -93.7%) vs baseline: +0.7%

Memory: ✅ 38.083MB (SLO: <39.000MB -2.4%) vs baseline: +4.9%

✅ replace_aspect

Time: ✅ 5.413µs (SLO: <10.000µs 📉 -45.9%) vs baseline: 📈 +13.4%

Memory: ✅ 38.044MB (SLO: <39.000MB -2.5%) vs baseline: +4.7%

✅ replace_noaspect

Time: ✅ 0.460µs (SLO: <10.000µs 📉 -95.4%) vs baseline: -0.8%

Memory: ✅ 38.063MB (SLO: <39.000MB -2.4%) vs baseline: +4.9%

✅ repr_aspect

Time: ✅ 0.909µs (SLO: <10.000µs 📉 -90.9%) vs baseline: -0.2%

Memory: ✅ 38.103MB (SLO: <39.000MB -2.3%) vs baseline: +4.9%

✅ repr_noaspect

Time: ✅ 0.421µs (SLO: <10.000µs 📉 -95.8%) vs baseline: +1.7%

Memory: ✅ 38.083MB (SLO: <39.000MB -2.4%) vs baseline: +4.8%

✅ rstrip_aspect

Time: ✅ 1.992µs (SLO: <20.000µs 📉 -90.0%) vs baseline: +3.9%

Memory: ✅ 38.024MB (SLO: <39.000MB -2.5%) vs baseline: +4.9%

✅ rstrip_noaspect

Time: ✅ 0.386µs (SLO: <10.000µs 📉 -96.1%) vs baseline: +1.0%

Memory: ✅ 38.083MB (SLO: <39.000MB -2.4%) vs baseline: +5.0%

✅ slice_aspect

Time: ✅ 0.498µs (SLO: <10.000µs 📉 -95.0%) vs baseline: +0.8%

Memory: ✅ 38.103MB (SLO: <39.000MB -2.3%) vs baseline: +4.8%

✅ slice_noaspect

Time: ✅ 0.450µs (SLO: <10.000µs 📉 -95.5%) vs baseline: ~same

Memory: ✅ 38.044MB (SLO: <39.000MB -2.5%) vs baseline: +4.6%

✅ stringio_aspect

Time: ✅ 1.697µs (SLO: <10.000µs 📉 -83.0%) vs baseline: 📈 +10.2%

Memory: ✅ 38.063MB (SLO: <39.000MB -2.4%) vs baseline: +5.0%

✅ stringio_noaspect

Time: ✅ 0.728µs (SLO: <10.000µs 📉 -92.7%) vs baseline: +1.8%

Memory: ✅ 38.063MB (SLO: <39.000MB -2.4%) vs baseline: +4.9%

✅ strip_aspect

Time: ✅ 2.551µs (SLO: <20.000µs 📉 -87.2%) vs baseline: 📈 +14.9%

Memory: ✅ 38.063MB (SLO: <39.000MB -2.4%) vs baseline: +4.7%

✅ strip_noaspect

Time: ✅ 0.384µs (SLO: <10.000µs 📉 -96.2%) vs baseline: +0.4%

Memory: ✅ 37.965MB (SLO: <39.000MB -2.7%) vs baseline: +4.4%

✅ swapcase_aspect

Time: ✅ 2.559µs (SLO: <10.000µs 📉 -74.4%) vs baseline: +5.4%

Memory: ✅ 38.083MB (SLO: <39.000MB -2.4%) vs baseline: +4.8%

✅ swapcase_noaspect

Time: ✅ 0.537µs (SLO: <10.000µs 📉 -94.6%) vs baseline: -0.1%

Memory: ✅ 38.044MB (SLO: <39.000MB -2.5%) vs baseline: +4.7%

✅ title_aspect

Time: ✅ 2.344µs (SLO: <10.000µs 📉 -76.6%) vs baseline: -1.1%

Memory: ✅ 38.083MB (SLO: <39.000MB -2.4%) vs baseline: +4.8%

✅ title_noaspect

Time: ✅ 0.498µs (SLO: <10.000µs 📉 -95.0%) vs baseline: -0.5%

Memory: ✅ 38.083MB (SLO: <39.000MB -2.4%) vs baseline: +4.9%

✅ translate_aspect

Time: ✅ 3.450µs (SLO: <10.000µs 📉 -65.5%) vs baseline: +7.4%

Memory: ✅ 38.044MB (SLO: <39.000MB -2.5%) vs baseline: +4.7%

✅ translate_noaspect

Time: ✅ 1.043µs (SLO: <10.000µs 📉 -89.6%) vs baseline: +0.2%

Memory: ✅ 38.103MB (SLO: <39.000MB -2.3%) vs baseline: +5.0%

✅ upper_aspect

Time: ✅ 2.356µs (SLO: <10.000µs 📉 -76.4%) vs baseline: +5.4%

Memory: ✅ 38.103MB (SLO: <39.000MB -2.3%) vs baseline: +5.2%

✅ upper_noaspect

Time: ✅ 0.375µs (SLO: <10.000µs 📉 -96.2%) vs baseline: -0.6%

Memory: ✅ 38.004MB (SLO: <39.000MB -2.6%) vs baseline: +4.7%

📈 iastaspectsospath - 24/24

✅ ospathbasename_aspect

Time: ✅ 4.313µs (SLO: <10.000µs 📉 -56.9%) vs baseline: +0.3%

Memory: ✅ 37.729MB (SLO: <39.000MB -3.3%) vs baseline: +4.9%

✅ ospathbasename_noaspect

Time: ✅ 1.090µs (SLO: <10.000µs 📉 -89.1%) vs baseline: ~same

Memory: ✅ 37.650MB (SLO: <39.000MB -3.5%) vs baseline: +4.9%

✅ ospathjoin_aspect

Time: ✅ 6.926µs (SLO: <10.000µs 📉 -30.7%) vs baseline: 📈 +13.5%

Memory: ✅ 37.690MB (SLO: <39.000MB -3.4%) vs baseline: +4.8%

✅ ospathjoin_noaspect

Time: ✅ 2.322µs (SLO: <10.000µs 📉 -76.8%) vs baseline: +0.5%

Memory: ✅ 37.670MB (SLO: <39.000MB -3.4%) vs baseline: +4.9%

✅ ospathnormcase_aspect

Time: ✅ 3.521µs (SLO: <10.000µs 📉 -64.8%) vs baseline: +0.7%

Memory: ✅ 37.709MB (SLO: <39.000MB -3.3%) vs baseline: +5.0%

✅ ospathnormcase_noaspect

Time: ✅ 0.570µs (SLO: <10.000µs 📉 -94.3%) vs baseline: ~same

Memory: ✅ 37.670MB (SLO: <39.000MB -3.4%) vs baseline: +4.9%

✅ ospathsplit_aspect

Time: ✅ 4.892µs (SLO: <10.000µs 📉 -51.1%) vs baseline: -0.4%

Memory: ✅ 37.631MB (SLO: <39.000MB -3.5%) vs baseline: +4.8%

✅ ospathsplit_noaspect

Time: ✅ 1.607µs (SLO: <10.000µs 📉 -83.9%) vs baseline: +0.5%

Memory: ✅ 37.690MB (SLO: <39.000MB -3.4%) vs baseline: +4.7%

✅ ospathsplitdrive_aspect

Time: ✅ 3.668µs (SLO: <10.000µs 📉 -63.3%) vs baseline: +0.3%

Memory: ✅ 37.690MB (SLO: <39.000MB -3.4%) vs baseline: +4.8%

✅ ospathsplitdrive_noaspect

Time: ✅ 0.699µs (SLO: <10.000µs 📉 -93.0%) vs baseline: -0.8%

Memory: ✅ 37.709MB (SLO: <39.000MB -3.3%) vs baseline: +4.9%

✅ ospathsplitext_aspect

Time: ✅ 4.537µs (SLO: <10.000µs 📉 -54.6%) vs baseline: -1.5%

Memory: ✅ 37.709MB (SLO: <39.000MB -3.3%) vs baseline: +4.9%

✅ ospathsplitext_noaspect

Time: ✅ 1.391µs (SLO: <10.000µs 📉 -86.1%) vs baseline: -0.6%

Memory: ✅ 37.690MB (SLO: <39.000MB -3.4%) vs baseline: +5.0%

📈 telemetryaddmetric - 30/30

✅ 1-count-metric-1-times

Time: ✅ 3.307µs (SLO: <20.000µs 📉 -83.5%) vs baseline: +6.4%

Memory: ✅ 32.126MB (SLO: <34.000MB -5.5%) vs baseline: +5.0%

✅ 1-count-metrics-100-times

Time: ✅ 213.621µs (SLO: <250.000µs 📉 -14.6%) vs baseline: +0.7%

Memory: ✅ 32.145MB (SLO: <34.000MB -5.5%) vs baseline: +5.0%

✅ 1-distribution-metric-1-times

Time: ✅ 3.201µs (SLO: <20.000µs 📉 -84.0%) vs baseline: +8.6%

Memory: ✅ 32.106MB (SLO: <34.000MB -5.6%) vs baseline: +4.8%

✅ 1-distribution-metrics-100-times

Time: ✅ 195.139µs (SLO: <220.000µs 📉 -11.3%) vs baseline: +1.4%

Memory: ✅ 32.067MB (SLO: <34.000MB -5.7%) vs baseline: +4.9%

✅ 1-gauge-metric-1-times

Time: ✅ 2.076µs (SLO: <20.000µs 📉 -89.6%) vs baseline: -1.4%

Memory: ✅ 32.126MB (SLO: <34.000MB -5.5%) vs baseline: +5.2%

✅ 1-gauge-metrics-100-times

Time: ✅ 126.386µs (SLO: <150.000µs 📉 -15.7%) vs baseline: +0.4%

Memory: ✅ 32.106MB (SLO: <34.000MB -5.6%) vs baseline: +4.9%

✅ 1-rate-metric-1-times

Time: ✅ 3.144µs (SLO: <20.000µs 📉 -84.3%) vs baseline: +1.2%

Memory: ✅ 32.145MB (SLO: <34.000MB -5.5%) vs baseline: +4.9%

✅ 1-rate-metrics-100-times

Time: ✅ 214.736µs (SLO: <250.000µs 📉 -14.1%) vs baseline: +1.0%

Memory: ✅ 32.145MB (SLO: <34.000MB -5.5%) vs baseline: +5.0%

✅ 100-count-metrics-100-times

Time: ✅ 21.761ms (SLO: <23.500ms -7.4%) vs baseline: +1.8%

Memory: ✅ 32.145MB (SLO: <34.000MB -5.5%) vs baseline: +4.9%

✅ 100-distribution-metrics-100-times

Time: ✅ 1.983ms (SLO: <2.250ms 📉 -11.9%) vs baseline: -0.2%

Memory: ✅ 32.165MB (SLO: <34.000MB -5.4%) vs baseline: +4.9%

✅ 100-gauge-metrics-100-times

Time: ✅ 1.295ms (SLO: <1.550ms 📉 -16.4%) vs baseline: +0.5%

Memory: ✅ 32.106MB (SLO: <34.000MB -5.6%) vs baseline: +4.7%

✅ 100-rate-metrics-100-times

Time: ✅ 2.217ms (SLO: <2.550ms 📉 -13.1%) vs baseline: +1.7%

Memory: ✅ 32.047MB (SLO: <34.000MB -5.7%) vs baseline: +4.8%

✅ flush-1-metric

Time: ✅ 4.687µs (SLO: <20.000µs 📉 -76.6%) vs baseline: 📈 +10.7%

Memory: ✅ 32.165MB (SLO: <34.000MB -5.4%) vs baseline: +5.2%

✅ flush-100-metrics

Time: ✅ 180.829µs (SLO: <250.000µs 📉 -27.7%) vs baseline: -0.4%

Memory: ✅ 32.106MB (SLO: <34.000MB -5.6%) vs baseline: +4.9%

✅ flush-1000-metrics

Time: ✅ 2.228ms (SLO: <2.500ms 📉 -10.9%) vs baseline: +0.8%

Memory: ✅ 32.873MB (SLO: <34.500MB -4.7%) vs baseline: +4.7%

🟡 Near SLO Breach (5 suites)

🟡 djangosimple - 30/30

✅ appsec

Time: ✅ 20.478ms (SLO: <22.300ms -8.2%) vs baseline: +0.3%

Memory: ✅ 65.488MB (SLO: <67.000MB -2.3%) vs baseline: +4.9%

✅ exception-replay-enabled

Time: ✅ 1.348ms (SLO: <1.450ms -7.1%) vs baseline: -0.3%

Memory: ✅ 64.591MB (SLO: <67.000MB -3.6%) vs baseline: +4.8%

✅ iast

Time: ✅ 20.482ms (SLO: <22.250ms -7.9%) vs baseline: +0.2%

Memory: ✅ 65.510MB (SLO: <67.000MB -2.2%) vs baseline: +4.9%

✅ profiler

Time: ✅ 15.323ms (SLO: <16.550ms -7.4%) vs baseline: +0.4%

Memory: ✅ 53.730MB (SLO: <54.500MB 🟡 -1.4%) vs baseline: +4.8%

✅ resource-renaming

Time: ✅ 20.513ms (SLO: <21.750ms -5.7%) vs baseline: -0.3%

Memory: ✅ 65.379MB (SLO: <67.000MB -2.4%) vs baseline: +4.8%

✅ span-code-origin

Time: ✅ 26.149ms (SLO: <28.200ms -7.3%) vs baseline: -0.1%

Memory: ✅ 67.480MB (SLO: <69.500MB -2.9%) vs baseline: +4.6%

✅ tracer

Time: ✅ 20.497ms (SLO: <21.750ms -5.8%) vs baseline: -0.2%

Memory: ✅ 65.487MB (SLO: <67.000MB -2.3%) vs baseline: +4.9%

✅ tracer-and-profiler

Time: ✅ 21.992ms (SLO: <23.500ms -6.4%) vs baseline: -0.2%

Memory: ✅ 66.578MB (SLO: <67.500MB 🟡 -1.4%) vs baseline: +4.9%

✅ tracer-dont-create-db-spans

Time: ✅ 19.332ms (SLO: <21.500ms 📉 -10.1%) vs baseline: ~same

Memory: ✅ 65.478MB (SLO: <66.000MB 🟡 -0.8%) vs baseline: +4.9%

✅ tracer-minimal

Time: ✅ 16.552ms (SLO: <17.500ms -5.4%) vs baseline: -0.5%

Memory: ✅ 65.467MB (SLO: <66.000MB 🟡 -0.8%) vs baseline: +4.7%

✅ tracer-native

Time: ✅ 20.452ms (SLO: <21.750ms -6.0%) vs baseline: -0.3%

Memory: ✅ 71.334MB (SLO: <72.500MB 🟡 -1.6%) vs baseline: +4.8%

✅ tracer-no-caches

Time: ✅ 18.438ms (SLO: <19.650ms -6.2%) vs baseline: ~same

Memory: ✅ 65.454MB (SLO: <67.000MB -2.3%) vs baseline: +4.9%

✅ tracer-no-databases

Time: ✅ 18.733ms (SLO: <20.100ms -6.8%) vs baseline: -0.2%

Memory: ✅ 65.299MB (SLO: <67.000MB -2.5%) vs baseline: +4.6%

✅ tracer-no-middleware

Time: ✅ 20.183ms (SLO: <21.500ms -6.1%) vs baseline: -0.1%

Memory: ✅ 65.467MB (SLO: <67.000MB -2.3%) vs baseline: +4.7%

✅ tracer-no-templates

Time: ✅ 20.297ms (SLO: <22.000ms -7.7%) vs baseline: ~same

Memory: ✅ 65.461MB (SLO: <67.000MB -2.3%) vs baseline: +4.9%

🟡 errortrackingdjangosimple - 6/6

✅ errortracking-enabled-all

Time: ✅ 18.288ms (SLO: <19.850ms -7.9%) vs baseline: +1.4%

Memory: ✅ 65.195MB (SLO: <66.500MB 🟡 -2.0%) vs baseline: +4.8%

✅ errortracking-enabled-user

Time: ✅ 18.262ms (SLO: <19.400ms -5.9%) vs baseline: +1.2%

Memory: ✅ 65.274MB (SLO: <66.500MB 🟡 -1.8%) vs baseline: +4.9%

✅ tracer-enabled

Time: ✅ 18.072ms (SLO: <19.450ms -7.1%) vs baseline: +0.3%

Memory: ✅ 65.235MB (SLO: <66.500MB 🟡 -1.9%) vs baseline: +4.9%

🟡 flasksimple - 18/18

✅ appsec-get

Time: ✅ 4.569ms (SLO: <4.750ms -3.8%) vs baseline: -0.3%

Memory: ✅ 62.010MB (SLO: <65.000MB -4.6%) vs baseline: +5.0%

✅ appsec-post

Time: ✅ 6.585ms (SLO: <6.750ms -2.4%) vs baseline: +0.2%

Memory: ✅ 61.991MB (SLO: <65.000MB -4.6%) vs baseline: +4.8%

✅ appsec-telemetry

Time: ✅ 4.555ms (SLO: <4.750ms -4.1%) vs baseline: -0.9%

Memory: ✅ 62.049MB (SLO: <65.000MB -4.5%) vs baseline: +4.9%

✅ debugger

Time: ✅ 1.887ms (SLO: <2.000ms -5.7%) vs baseline: +1.3%

Memory: ✅ 45.554MB (SLO: <47.000MB -3.1%) vs baseline: +5.2%

✅ iast-get

Time: ✅ 1.861ms (SLO: <2.000ms -6.9%) vs baseline: -0.3%

Memory: ✅ 42.408MB (SLO: <49.000MB 📉 -13.5%) vs baseline: +4.9%

✅ profiler

Time: ✅ 1.906ms (SLO: <2.100ms -9.3%) vs baseline: -0.4%

Memory: ✅ 46.478MB (SLO: <47.000MB 🟡 -1.1%) vs baseline: +4.8%

✅ resource-renaming

Time: ✅ 3.381ms (SLO: <3.650ms -7.4%) vs baseline: -0.2%

Memory: ✅ 52.199MB (SLO: <53.500MB -2.4%) vs baseline: +4.8%

✅ tracer

Time: ✅ 3.363ms (SLO: <3.650ms -7.9%) vs baseline: -0.7%

Memory: ✅ 52.258MB (SLO: <53.500MB -2.3%) vs baseline: +4.9%

✅ tracer-native

Time: ✅ 3.370ms (SLO: <3.650ms -7.7%) vs baseline: -0.1%

Memory: ✅ 58.260MB (SLO: <60.000MB -2.9%) vs baseline: +4.8%

🟡 otelspan - 22/22

✅ add-event

Time: ✅ 42.662ms (SLO: <47.150ms -9.5%) vs baseline: +0.7%

Memory: ✅ 44.463MB (SLO: <47.000MB -5.4%) vs baseline: +4.9%

✅ add-metrics

Time: ✅ 318.139ms (SLO: <344.800ms -7.7%) vs baseline: ~same

Memory: ✅ 596.460MB (SLO: <600.000MB 🟡 -0.6%) vs baseline: +5.0%

✅ add-tags

Time: ✅ 291.973ms (SLO: <314.000ms -7.0%) vs baseline: +1.9%

Memory: ✅ 597.205MB (SLO: <600.000MB 🟡 -0.5%) vs baseline: +4.9%

✅ get-context

Time: ✅ 80.926ms (SLO: <92.350ms 📉 -12.4%) vs baseline: +0.3%

Memory: ✅ 39.910MB (SLO: <46.500MB 📉 -14.2%) vs baseline: +4.9%

✅ is-recording

Time: ✅ 38.974ms (SLO: <44.500ms 📉 -12.4%) vs baseline: -0.2%

Memory: ✅ 43.931MB (SLO: <47.500MB -7.5%) vs baseline: +4.8%

✅ record-exception

Time: ✅ 58.884ms (SLO: <67.650ms 📉 -13.0%) vs baseline: ~same

Memory: ✅ 40.275MB (SLO: <47.000MB 📉 -14.3%) vs baseline: +4.7%

✅ set-status

Time: ✅ 45.646ms (SLO: <50.400ms -9.4%) vs baseline: +1.9%

Memory: ✅ 43.854MB (SLO: <47.000MB -6.7%) vs baseline: +4.6%

✅ start

Time: ✅ 38.246ms (SLO: <43.450ms 📉 -12.0%) vs baseline: +0.2%

Memory: ✅ 43.986MB (SLO: <47.000MB -6.4%) vs baseline: +5.0%

✅ start-finish

Time: ✅ 82.675ms (SLO: <88.000ms -6.1%) vs baseline: -0.2%

Memory: ✅ 34.544MB (SLO: <46.500MB 📉 -25.7%) vs baseline: +4.9%

✅ start-finish-telemetry

Time: ✅ 85.648ms (SLO: <89.000ms -3.8%) vs baseline: +1.7%

Memory: ✅ 34.524MB (SLO: <46.500MB 📉 -25.8%) vs baseline: +4.7%

✅ update-name

Time: ✅ 40.183ms (SLO: <45.150ms 📉 -11.0%) vs baseline: +0.3%

Memory: ✅ 44.210MB (SLO: <47.000MB -5.9%) vs baseline: +4.8%

🟡 span - 26/26

✅ add-event

Time: ✅ 21.104ms (SLO: <22.500ms -6.2%) vs baseline: +2.6%

Memory: ✅ 50.354MB (SLO: <53.000MB -5.0%) vs baseline: +4.9%

✅ add-metrics

Time: ✅ 91.053ms (SLO: <93.500ms -2.6%) vs baseline: -0.3%

Memory: ✅ 660.808MB (SLO: <961.000MB 📉 -31.2%) vs baseline: +4.8%

✅ add-tags

Time: ✅ 149.171ms (SLO: <155.000ms -3.8%) vs baseline: +0.6%

Memory: ✅ 661.512MB (SLO: <962.500MB 📉 -31.3%) vs baseline: +4.8%

✅ get-context

Time: ✅ 19.425ms (SLO: <20.500ms -5.2%) vs baseline: -0.4%

Memory: ✅ 49.113MB (SLO: <53.000MB -7.3%) vs baseline: +4.8%

✅ is-recording

Time: ✅ 19.685ms (SLO: <20.500ms -4.0%) vs baseline: -0.2%

Memory: ✅ 49.189MB (SLO: <53.000MB -7.2%) vs baseline: +4.8%

✅ record-exception

Time: ✅ 38.221ms (SLO: <40.000ms -4.4%) vs baseline: -0.4%

Memory: ✅ 42.752MB (SLO: <53.000MB 📉 -19.3%) vs baseline: +4.9%

✅ set-status

Time: ✅ 21.354ms (SLO: <22.000ms -2.9%) vs baseline: +0.7%

Memory: ✅ 49.217MB (SLO: <53.000MB -7.1%) vs baseline: +5.1%

✅ start

Time: ✅ 19.202ms (SLO: <20.500ms -6.3%) vs baseline: -0.6%

Memory: ✅ 49.182MB (SLO: <53.000MB -7.2%) vs baseline: +4.7%

✅ start-finish

Time: ✅ 51.547ms (SLO: <52.500ms 🟡 -1.8%) vs baseline: ~same

Memory: ✅ 32.126MB (SLO: <34.000MB -5.5%) vs baseline: +4.5%

✅ start-finish-telemetry

Time: ✅ 52.822ms (SLO: <54.500ms -3.1%) vs baseline: ~same

Memory: ✅ 32.204MB (SLO: <34.000MB -5.3%) vs baseline: +5.2%

✅ start-finish-traceid128

Time: ✅ 54.993ms (SLO: <56.000ms 🟡 -1.8%) vs baseline: +0.3%

Memory: ✅ 32.126MB (SLO: <34.000MB -5.5%) vs baseline: +4.9%

✅ start-traceid128

Time: ✅ 19.726ms (SLO: <22.500ms 📉 -12.3%) vs baseline: -0.4%

Memory: ✅ 49.110MB (SLO: <53.000MB -7.3%) vs baseline: +5.0%

✅ update-name

Time: ✅ 20.267ms (SLO: <22.000ms -7.9%) vs baseline: +0.4%

Memory: ✅ 49.752MB (SLO: <53.000MB -6.1%) vs baseline: +4.7%

⚠️ Unstable Tests (1 suite)

⚠️

coreapiscenario - 10/10 (1 unstable)

⚠️ context_with_data_listeners

Time: ⚠️ 13.265µs (SLO: <20.000µs 📉 -33.7%) vs baseline: ~same

Memory: ✅ 32.165MB (SLO: <33.500MB -4.0%) vs baseline: +5.0%

✅ context_with_data_no_listeners

Time: ✅ 3.269µs (SLO: <10.000µs 📉 -67.3%) vs baseline: -1.0%

Memory: ✅ 32.067MB (SLO: <33.500MB -4.3%) vs baseline: +4.8%

✅ get_item_exists

Time: ✅ 0.585µs (SLO: <10.000µs 📉 -94.2%) vs baseline: +0.5%

Memory: ✅ 32.126MB (SLO: <33.500MB -4.1%) vs baseline: +4.9%

✅ get_item_missing

Time: ✅ 0.641µs (SLO: <10.000µs 📉 -93.6%) vs baseline: +0.5%

Memory: ✅ 32.126MB (SLO: <33.500MB -4.1%) vs baseline: +4.9%

✅ set_item

Time: ✅ 24.064µs (SLO: <30.000µs 📉 -19.8%) vs baseline: +0.1%

Memory: ✅ 32.086MB (SLO: <33.500MB -4.2%) vs baseline: +4.7%

✅ All Tests Passing (15 suites)

✅ errortrackingflasksqli - 6/6

✅ errortracking-enabled-all

Time: ✅ 2.084ms (SLO: <2.300ms -9.4%) vs baseline: +0.5%

Memory: ✅ 52.160MB (SLO: <53.500MB -2.5%) vs baseline: +5.1%

✅ errortracking-enabled-user

Time: ✅ 2.096ms (SLO: <2.250ms -6.9%) vs baseline: +0.5%

Memory: ✅ 52.062MB (SLO: <53.500MB -2.7%) vs baseline: +4.7%

✅ tracer-enabled

Time: ✅ 2.085ms (SLO: <2.300ms -9.3%) vs baseline: +0.2%

Memory: ✅ 52.160MB (SLO: <53.500MB -2.5%) vs baseline: +5.2%

✅ flasksqli - 6/6

✅ appsec-enabled

Time: ✅ 3.921ms (SLO: <4.200ms -6.6%) vs baseline: +0.1%

Memory: ✅ 62.148MB (SLO: <66.000MB -5.8%) vs baseline: +4.6%

✅ iast-enabled

Time: ✅ 2.450ms (SLO: <2.800ms 📉 -12.5%) vs baseline: -0.1%

Memory: ✅ 58.668MB (SLO: <60.000MB -2.2%) vs baseline: +4.9%

✅ tracer-enabled

Time: ✅ 2.066ms (SLO: <2.250ms -8.2%) vs baseline: ~same

Memory: ✅ 52.199MB (SLO: <54.500MB -4.2%) vs baseline: +5.1%

✅ httppropagationextract - 60/60

✅ all_styles_all_headers

Time: ✅ 82.545µs (SLO: <100.000µs 📉 -17.5%) vs baseline: +0.9%

Memory: ✅ 32.067MB (SLO: <33.500MB -4.3%) vs baseline: +4.6%

✅ b3_headers

Time: ✅ 14.337µs (SLO: <20.000µs 📉 -28.3%) vs baseline: +1.7%

Memory: ✅ 32.126MB (SLO: <33.500MB -4.1%) vs baseline: +4.5%

✅ b3_single_headers

Time: ✅ 13.410µs (SLO: <20.000µs 📉 -33.0%) vs baseline: +1.2%

Memory: ✅ 32.145MB (SLO: <33.500MB -4.0%) vs baseline: +4.7%

✅ datadog_tracecontext_tracestate_not_propagated_on_trace_id_no_match

Time: ✅ 64.722µs (SLO: <80.000µs 📉 -19.1%) vs baseline: +1.2%

Memory: ✅ 32.126MB (SLO: <33.500MB -4.1%) vs baseline: +4.6%

✅ datadog_tracecontext_tracestate_propagated_on_trace_id_match

Time: ✅ 66.481µs (SLO: <80.000µs 📉 -16.9%) vs baseline: +0.5%

Memory: ✅ 32.165MB (SLO: <33.500MB -4.0%) vs baseline: +4.9%

✅ empty_headers

Time: ✅ 1.598µs (SLO: <10.000µs 📉 -84.0%) vs baseline: -0.1%

Memory: ✅ 32.165MB (SLO: <33.500MB -4.0%) vs baseline: +4.9%

✅ full_t_id_datadog_headers

Time: ✅ 22.828µs (SLO: <30.000µs 📉 -23.9%) vs baseline: -0.5%

Memory: ✅ 32.106MB (SLO: <33.500MB -4.2%) vs baseline: +4.6%

✅ invalid_priority_header

Time: ✅ 6.468µs (SLO: <10.000µs 📉 -35.3%) vs baseline: ~same

Memory: ✅ 32.126MB (SLO: <33.500MB -4.1%) vs baseline: +4.7%

✅ invalid_span_id_header

Time: ✅ 6.510µs (SLO: <10.000µs 📉 -34.9%) vs baseline: -0.9%

Memory: ✅ 32.204MB (SLO: <33.500MB -3.9%) vs baseline: +4.9%

✅ invalid_tags_header

Time: ✅ 6.521µs (SLO: <10.000µs 📉 -34.8%) vs baseline: +0.4%

Memory: ✅ 32.165MB (SLO: <33.500MB -4.0%) vs baseline: +4.7%

✅ invalid_trace_id_header

Time: ✅ 6.535µs (SLO: <10.000µs 📉 -34.7%) vs baseline: +0.3%

Memory: ✅ 32.126MB (SLO: <33.500MB -4.1%) vs baseline: +4.7%

✅ large_header_no_matches

Time: ✅ 27.569µs (SLO: <30.000µs -8.1%) vs baseline: -0.1%

Memory: ✅ 32.165MB (SLO: <33.500MB -4.0%) vs baseline: +5.0%

✅ large_valid_headers_all

Time: ✅ 28.610µs (SLO: <40.000µs 📉 -28.5%) vs baseline: ~same

Memory: ✅ 32.204MB (SLO: <33.500MB -3.9%) vs baseline: +5.0%

✅ medium_header_no_matches

Time: ✅ 9.822µs (SLO: <20.000µs 📉 -50.9%) vs baseline: -0.2%

Memory: ✅ 32.145MB (SLO: <33.500MB -4.0%) vs baseline: +4.8%

✅ medium_valid_headers_all

Time: ✅ 11.178µs (SLO: <20.000µs 📉 -44.1%) vs baseline: -0.5%

Memory: ✅ 32.185MB (SLO: <33.500MB -3.9%) vs baseline: +4.9%

✅ none_propagation_style

Time: ✅ 1.700µs (SLO: <10.000µs 📉 -83.0%) vs baseline: -0.1%

Memory: ✅ 32.224MB (SLO: <33.500MB -3.8%) vs baseline: +4.9%

✅ tracecontext_headers

Time: ✅ 34.466µs (SLO: <40.000µs 📉 -13.8%) vs baseline: +0.3%

Memory: ✅ 32.126MB (SLO: <33.500MB -4.1%) vs baseline: +4.7%

✅ valid_headers_all

Time: ✅ 6.493µs (SLO: <10.000µs 📉 -35.1%) vs baseline: -0.3%

Memory: ✅ 32.145MB (SLO: <33.500MB -4.0%) vs baseline: +4.9%

✅ valid_headers_basic

Time: ✅ 6.085µs (SLO: <10.000µs 📉 -39.2%) vs baseline: -0.1%

Memory: ✅ 32.106MB (SLO: <33.500MB -4.2%) vs baseline: +4.6%

✅ wsgi_empty_headers

Time: ✅ 1.605µs (SLO: <10.000µs 📉 -84.0%) vs baseline: -0.3%

Memory: ✅ 32.126MB (SLO: <33.500MB -4.1%) vs baseline: +4.7%

✅ wsgi_invalid_priority_header

Time: ✅ 6.565µs (SLO: <10.000µs 📉 -34.4%) vs baseline: +0.7%

Memory: ✅ 32.126MB (SLO: <33.500MB -4.1%) vs baseline: +4.6%

✅ wsgi_invalid_span_id_header

Time: ✅ 1.591µs (SLO: <10.000µs 📉 -84.1%) vs baseline: -0.8%

Memory: ✅ 32.126MB (SLO: <33.500MB -4.1%) vs baseline: +4.6%

✅ wsgi_invalid_tags_header

Time: ✅ 6.577µs (SLO: <10.000µs 📉 -34.2%) vs baseline: +0.6%

Memory: ✅ 32.106MB (SLO: <33.500MB -4.2%) vs baseline: +4.8%

✅ wsgi_invalid_trace_id_header

Time: ✅ 6.582µs (SLO: <10.000µs 📉 -34.2%) vs baseline: +0.1%

Memory: ✅ 32.145MB (SLO: <33.500MB -4.0%) vs baseline: +5.2%

✅ wsgi_large_header_no_matches

Time: ✅ 28.717µs (SLO: <40.000µs 📉 -28.2%) vs baseline: ~same

Memory: ✅ 32.204MB (SLO: <33.500MB -3.9%) vs baseline: +5.0%

✅ wsgi_large_valid_headers_all

Time: ✅ 29.883µs (SLO: <40.000µs 📉 -25.3%) vs baseline: +0.6%

Memory: ✅ 32.224MB (SLO: <33.500MB -3.8%) vs baseline: +4.9%

✅ wsgi_medium_header_no_matches

Time: ✅ 10.096µs (SLO: <20.000µs 📉 -49.5%) vs baseline: +0.4%

Memory: ✅ 32.204MB (SLO: <33.500MB -3.9%) vs baseline: +5.0%

✅ wsgi_medium_valid_headers_all

Time: ✅ 11.558µs (SLO: <20.000µs 📉 -42.2%) vs baseline: ~same

Memory: ✅ 32.126MB (SLO: <33.500MB -4.1%) vs baseline: +4.7%

✅ wsgi_valid_headers_all

Time: ✅ 6.561µs (SLO: <10.000µs 📉 -34.4%) vs baseline: -0.1%

Memory: ✅ 32.165MB (SLO: <33.500MB -4.0%) vs baseline: +4.7%

✅ wsgi_valid_headers_basic

Time: ✅ 6.229µs (SLO: <10.000µs 📉 -37.7%) vs baseline: +2.0%

Memory: ✅ 32.224MB (SLO: <33.500MB -3.8%) vs baseline: +5.1%

✅ httppropagationinject - 16/16

✅ ids_only

Time: ✅ 21.760µs (SLO: <30.000µs 📉 -27.5%) vs baseline: +0.5%

Memory: ✅ 32.165MB (SLO: <33.500MB -4.0%) vs baseline: +4.8%

✅ with_all

Time: ✅ 28.888µs (SLO: <40.000µs 📉 -27.8%) vs baseline: +0.1%

Memory: ✅ 32.145MB (SLO: <33.500MB -4.0%) vs baseline: +4.7%

✅ with_dd_origin

Time: ✅ 25.410µs (SLO: <30.000µs 📉 -15.3%) vs baseline: +0.3%

Memory: ✅ 32.165MB (SLO: <33.500MB -4.0%) vs baseline: +4.8%

✅ with_priority_and_origin

Time: ✅ 24.674µs (SLO: <40.000µs 📉 -38.3%) vs baseline: -0.6%

Memory: ✅ 32.126MB (SLO: <33.500MB -4.1%) vs baseline: +4.5%

✅ with_sampling_priority

Time: ✅ 21.608µs (SLO: <30.000µs 📉 -28.0%) vs baseline: -0.1%

Memory: ✅ 32.185MB (SLO: <33.500MB -3.9%) vs baseline: +5.1%

✅ with_tags

Time: ✅ 26.934µs (SLO: <40.000µs 📉 -32.7%) vs baseline: -0.2%

Memory: ✅ 32.185MB (SLO: <33.500MB -3.9%) vs baseline: +4.7%

✅ with_tags_invalid

Time: ✅ 28.420µs (SLO: <40.000µs 📉 -29.0%) vs baseline: +0.4%

Memory: ✅ 32.185MB (SLO: <33.500MB -3.9%) vs baseline: +4.7%

✅ with_tags_max_size

Time: ✅ 27.640µs (SLO: <40.000µs 📉 -30.9%) vs baseline: +1.0%

Memory: ✅ 32.165MB (SLO: <33.500MB -4.0%) vs baseline: +4.9%

✅ iast_aspects - 40/40

✅ re_expand_aspect

Time: ✅ 32.072µs (SLO: <40.000µs 📉 -19.8%) vs baseline: +0.5%

Memory: ✅ 37.709MB (SLO: <39.000MB -3.3%) vs baseline: +5.1%

✅ re_expand_noaspect

Time: ✅ 28.745µs (SLO: <40.000µs 📉 -28.1%) vs baseline: +1.9%

Memory: ✅ 37.650MB (SLO: <39.000MB -3.5%) vs baseline: +4.7%

✅ re_findall_aspect

Time: ✅ 2.895µs (SLO: <10.000µs 📉 -71.0%) vs baseline: -0.2%

Memory: ✅ 37.690MB (SLO: <39.000MB -3.4%) vs baseline: +4.9%

✅ re_findall_noaspect

Time: ✅ 1.411µs (SLO: <10.000µs 📉 -85.9%) vs baseline: -0.8%

Memory: ✅ 37.690MB (SLO: <39.000MB -3.4%) vs baseline: +4.9%

✅ re_finditer_aspect

Time: ✅ 4.499µs (SLO: <10.000µs 📉 -55.0%) vs baseline: +0.4%

Memory: ✅ 37.670MB (SLO: <39.000MB -3.4%) vs baseline: +4.9%

✅ re_finditer_noaspect

Time: ✅ 1.409µs (SLO: <10.000µs 📉 -85.9%) vs baseline: -0.5%

Memory: ✅ 37.631MB (SLO: <39.000MB -3.5%) vs baseline: +4.9%

✅ re_fullmatch_aspect

Time: ✅ 2.665µs (SLO: <10.000µs 📉 -73.3%) vs baseline: -0.6%

Memory: ✅ 37.670MB (SLO: <39.000MB -3.4%) vs baseline: +4.7%

✅ re_fullmatch_noaspect

Time: ✅ 1.306µs (SLO: <10.000µs 📉 -86.9%) vs baseline: ~same

Memory: ✅ 37.670MB (SLO: <39.000MB -3.4%) vs baseline: +4.9%

✅ re_group_aspect

Time: ✅ 2.962µs (SLO: <10.000µs 📉 -70.4%) vs baseline: +0.2%

Memory: ✅ 37.729MB (SLO: <39.000MB -3.3%) vs baseline: +5.0%

✅ re_group_noaspect

Time: ✅ 1.610µs (SLO: <10.000µs 📉 -83.9%) vs baseline: -0.2%

Memory: ✅ 37.670MB (SLO: <39.000MB -3.4%) vs baseline: +4.9%

✅ re_groups_aspect

Time: ✅ 3.106µs (SLO: <10.000µs 📉 -68.9%) vs baseline: +1.2%

Memory: ✅ 37.650MB (SLO: <39.000MB -3.5%) vs baseline: +4.9%

✅ re_groups_noaspect

Time: ✅ 1.705µs (SLO: <10.000µs 📉 -83.0%) vs baseline: +0.2%

Memory: ✅ 37.650MB (SLO: <39.000MB -3.5%) vs baseline: +4.9%

✅ re_match_aspect

Time: ✅ 2.727µs (SLO: <10.000µs 📉 -72.7%) vs baseline: +0.4%

Memory: ✅ 37.650MB (SLO: <39.000MB -3.5%) vs baseline: +4.6%

✅ re_match_noaspect

Time: ✅ 1.312µs (SLO: <10.000µs 📉 -86.9%) vs baseline: +0.6%

Memory: ✅ 37.729MB (SLO: <39.000MB -3.3%) vs baseline: +4.9%

✅ re_search_aspect

Time: ✅ 2.577µs (SLO: <10.000µs 📉 -74.2%) vs baseline: +0.2%

Memory: ✅ 37.690MB (SLO: <39.000MB -3.4%) vs baseline: +4.8%

✅ re_search_noaspect

Time: ✅ 1.203µs (SLO: <10.000µs 📉 -88.0%) vs baseline: -1.5%

Memory: ✅ 37.650MB (SLO: <39.000MB -3.5%) vs baseline: +4.7%

✅ re_sub_aspect

Time: ✅ 3.440µs (SLO: <10.000µs 📉 -65.6%) vs baseline: +0.2%

Memory: ✅ 37.670MB (SLO: <39.000MB -3.4%) vs baseline: +4.9%

✅ re_sub_noaspect

Time: ✅ 1.543µs (SLO: <10.000µs 📉 -84.6%) vs baseline: +0.2%

Memory: ✅ 37.729MB (SLO: <39.000MB -3.3%) vs baseline: +5.0%

✅ re_subn_aspect

Time: ✅ 3.672µs (SLO: <10.000µs 📉 -63.3%) vs baseline: +0.8%

Memory: ✅ 37.631MB (SLO: <39.000MB -3.5%) vs baseline: +4.6%

✅ re_subn_noaspect

Time: ✅ 1.641µs (SLO: <10.000µs 📉 -83.6%) vs baseline: +0.8%

Memory: ✅ 37.650MB (SLO: <39.000MB -3.5%) vs baseline: +4.9%

✅ iastaspectssplit - 12/12

✅ rsplit_aspect

Time: ✅ 1.436µs (SLO: <10.000µs 📉 -85.6%) vs baseline: +1.8%

Memory: ✅ 37.690MB (SLO: <39.000MB -3.4%) vs baseline: +5.0%

✅ rsplit_noaspect

Time: ✅ 0.579µs (SLO: <10.000µs 📉 -94.2%) vs baseline: -0.3%

Memory: ✅ 37.709MB (SLO: <39.000MB -3.3%) vs baseline: +4.9%

✅ split_aspect

Time: ✅ 1.432µs (SLO: <10.000µs 📉 -85.7%) vs baseline: +1.5%

Memory: ✅ 37.650MB (SLO: <39.000MB -3.5%) vs baseline: +4.7%

✅ split_noaspect

Time: ✅ 0.570µs (SLO: <10.000µs 📉 -94.3%) vs baseline: +0.2%

Memory: ✅ 37.690MB (SLO: <39.000MB -3.4%) vs baseline: +4.9%

✅ splitlines_aspect

Time: ✅ 1.423µs (SLO: <10.000µs 📉 -85.8%) vs baseline: +1.1%

Memory: ✅ 37.709MB (SLO: <39.000MB -3.3%) vs baseline: +4.9%

✅ splitlines_noaspect

Time: ✅ 0.588µs (SLO: <10.000µs 📉 -94.1%) vs baseline: +1.1%

Memory: ✅ 37.650MB (SLO: <39.000MB -3.5%) vs baseline: +4.7%

✅ iastpropagation - 2/2

✅ no-propagation

Time: ✅ 49.373µs (SLO: <60.000µs 📉 -17.7%) vs baseline: ~same

Memory: ✅ 37.729MB (SLO: <39.000MB -3.3%) vs baseline: +4.8%

✅ otelsdkspan - 24/24

✅ add-event

Time: ✅ 40.329ms (SLO: <42.000ms -4.0%) vs baseline: +0.2%

Memory: ✅ 34.505MB (SLO: <39.000MB 📉 -11.5%) vs baseline: +4.7%

✅ add-link

Time: ✅ 36.446ms (SLO: <38.550ms -5.5%) vs baseline: +0.3%

Memory: ✅ 34.564MB (SLO: <39.000MB 📉 -11.4%) vs baseline: +5.0%

✅ add-metrics

Time: ✅ 219.615ms (SLO: <232.000ms -5.3%) vs baseline: +1.1%

Memory: ✅ 34.839MB (SLO: <39.000MB 📉 -10.7%) vs baseline: +4.6%

✅ add-tags

Time: ✅ 213.774ms (SLO: <221.600ms -3.5%) vs baseline: +1.4%

Memory: ✅ 34.564MB (SLO: <39.000MB 📉 -11.4%) vs baseline: +5.1%

✅ get-context

Time: ✅ 29.116ms (SLO: <31.300ms -7.0%) vs baseline: +0.3%

Memory: ✅ 34.544MB (SLO: <39.000MB 📉 -11.4%) vs baseline: +4.0%

✅ is-recording

Time: ✅ 28.968ms (SLO: <31.000ms -6.6%) vs baseline: +0.2%

Memory: ✅ 34.583MB (SLO: <39.000MB 📉 -11.3%) vs baseline: +4.4%

✅ record-exception

Time: ✅ 63.181ms (SLO: <65.850ms -4.1%) vs baseline: ~same

Memory: ✅ 34.898MB (SLO: <39.000MB 📉 -10.5%) vs baseline: +4.8%

✅ set-status

Time: ✅ 31.935ms (SLO: <34.150ms -6.5%) vs baseline: -0.2%

Memory: ✅ 34.642MB (SLO: <39.000MB 📉 -11.2%) vs baseline: +4.5%

✅ start

Time: ✅ 28.707ms (SLO: <30.150ms -4.8%) vs baseline: +0.3%

Memory: ✅ 34.505MB (SLO: <39.000MB 📉 -11.5%) vs baseline: +4.3%

✅ start-finish

Time: ✅ 33.880ms (SLO: <35.350ms -4.2%) vs baseline: +0.3%

Memory: ✅ 34.564MB (SLO: <39.000MB 📉 -11.4%) vs baseline: +4.3%

✅ start-finish-telemetry

Time: ✅ 33.788ms (SLO: <35.450ms -4.7%) vs baseline: -0.3%

Memory: ✅ 34.603MB (SLO: <39.000MB 📉 -11.3%) vs baseline: +4.7%

✅ update-name

Time: ✅ 31.177ms (SLO: <33.400ms -6.7%) vs baseline: +0.4%

Memory: ✅ 34.642MB (SLO: <39.000MB 📉 -11.2%) vs baseline: +4.0%

✅ packagespackageforrootmodulemapping - 4/4

✅ cache_off

Time: ✅ 342.608ms (SLO: <354.300ms -3.3%) vs baseline: ~same

Memory: ✅ 38.240MB (SLO: <40.000MB -4.4%) vs baseline: +6.1%

✅ cache_on

Time: ✅ 0.383µs (SLO: <10.000µs 📉 -96.2%) vs baseline: -0.6%

Memory: ✅ 36.389MB (SLO: <39.000MB -6.7%) vs baseline: +5.5%

✅ packagesupdateimporteddependencies - 24/24

✅ import_many

Time: ✅ 154.654µs (SLO: <170.000µs -9.0%) vs baseline: -0.3%

Memory: ✅ 37.300MB (SLO: <38.500MB -3.1%) vs baseline: +4.8%

✅ import_many_cached

Time: ✅ 122.740µs (SLO: <130.000µs -5.6%) vs baseline: +1.3%

Memory: ✅ 37.231MB (SLO: <38.500MB -3.3%) vs baseline: +4.4%

✅ import_many_stdlib

Time: ✅ 1.626ms (SLO: <1.750ms -7.1%) vs baseline: -0.7%

Memory: ✅ 37.486MB (SLO: <38.500MB -2.6%) vs baseline: +5.0%

✅ import_many_stdlib_cached

Time: ✅ 0.974ms (SLO: <1.100ms 📉 -11.4%) vs baseline: -0.5%

Memory: ✅ 37.413MB (SLO: <38.500MB -2.8%) vs baseline: +4.7%

✅ import_many_unknown

Time: ✅ 828.324µs (SLO: <890.000µs -6.9%) vs baseline: -0.2%

Memory: ✅ 37.480MB (SLO: <38.500MB -2.7%) vs baseline: +4.7%

✅ import_many_unknown_cached

Time: ✅ 785.622µs (SLO: <870.000µs -9.7%) vs baseline: -0.5%

Memory: ✅ 37.496MB (SLO: <38.500MB -2.6%) vs baseline: +5.1%

✅ import_one

Time: ✅ 19.685µs (SLO: <30.000µs 📉 -34.4%) vs baseline: -0.7%

Memory: ✅ 37.279MB (SLO: <39.000MB -4.4%) vs baseline: +4.9%

✅ import_one_cache

Time: ✅ 6.293µs (SLO: <10.000µs 📉 -37.1%) vs baseline: -0.9%

Memory: ✅ 37.267MB (SLO: <38.500MB -3.2%) vs baseline: +4.7%

✅ import_one_stdlib

Time: ✅ 18.648µs (SLO: <20.000µs -6.8%) vs baseline: -0.5%

Memory: ✅ 37.293MB (SLO: <38.500MB -3.1%) vs baseline: +4.7%

✅ import_one_stdlib_cache

Time: ✅ 6.297µs (SLO: <10.000µs 📉 -37.0%) vs baseline: +0.3%

Memory: ✅ 37.305MB (SLO: <38.500MB -3.1%) vs baseline: +4.7%

✅ import_one_unknown

Time: ✅ 45.157µs (SLO: <50.000µs -9.7%) vs baseline: -0.9%

Memory: ✅ 37.271MB (SLO: <38.500MB -3.2%) vs baseline: +4.7%

✅ import_one_unknown_cache

Time: ✅ 6.290µs (SLO: <10.000µs 📉 -37.1%) vs baseline: -0.2%

Memory: ✅ 37.266MB (SLO: <38.500MB -3.2%) vs baseline: +4.7%

✅ ratelimiter - 12/12

✅ defaults

Time: ✅ 2.358µs (SLO: <10.000µs 📉 -76.4%) vs baseline: -0.2%

Memory: ✅ 31.772MB (SLO: <34.000MB -6.6%) vs baseline: +4.9%

✅ high_rate_limit

Time: ✅ 2.409µs (SLO: <10.000µs 📉 -75.9%) vs baseline: -0.6%

Memory: ✅ 31.752MB (SLO: <34.000MB -6.6%) vs baseline: +4.9%

✅ long_window

Time: ✅ 2.363µs (SLO: <10.000µs 📉 -76.4%) vs baseline: ~same

Memory: ✅ 31.792MB (SLO: <34.000MB -6.5%) vs baseline: +4.9%

✅ low_rate_limit

Time: ✅ 2.361µs (SLO: <10.000µs 📉 -76.4%) vs baseline: -0.2%

Memory: ✅ 31.811MB (SLO: <34.000MB -6.4%) vs baseline: +5.1%

✅ no_rate_limit

Time: ✅ 0.827µs (SLO: <10.000µs 📉 -91.7%) vs baseline: +0.3%

Memory: ✅ 31.772MB (SLO: <34.000MB -6.6%) vs baseline: +4.8%

✅ short_window

Time: ✅ 2.488µs (SLO: <10.000µs 📉 -75.1%) vs baseline: -0.4%

Memory: ✅ 31.811MB (SLO: <34.000MB -6.4%) vs baseline: +4.8%

✅ recursivecomputation - 8/8

✅ deep

Time: ✅ 308.971ms (SLO: <320.950ms -3.7%) vs baseline: -0.2%

Memory: ✅ 32.912MB (SLO: <34.500MB -4.6%) vs baseline: +4.7%

✅ deep-profiled

Time: ✅ 326.836ms (SLO: <359.150ms -9.0%) vs baseline: -0.3%

Memory: ✅ 37.257MB (SLO: <39.000MB -4.5%) vs baseline: +4.8%

✅ medium

Time: ✅ 7.087ms (SLO: <7.400ms -4.2%) vs baseline: +0.1%

Memory: ✅ 32.086MB (SLO: <34.000MB -5.6%) vs baseline: +4.8%

✅ shallow

Time: ✅ 0.957ms (SLO: <1.050ms -8.9%) vs baseline: +0.2%

Memory: ✅ 32.106MB (SLO: <34.000MB -5.6%) vs baseline: +4.9%

✅ samplingrules - 8/8

✅ average_match

Time: ✅ 148.178µs (SLO: <290.000µs 📉 -48.9%) vs baseline: +0.5%

Memory: ✅ 32.086MB (SLO: <34.000MB -5.6%) vs baseline: +4.5%

✅ high_match

Time: ✅ 195.973µs (SLO: <480.000µs 📉 -59.2%) vs baseline: +0.6%

Memory: ✅ 32.126MB (SLO: <34.000MB -5.5%) vs baseline: +4.8%

✅ low_match

Time: ✅ 100.106µs (SLO: <120.000µs 📉 -16.6%) vs baseline: -0.3%

Memory: ✅ 632.636MB (SLO: <700.000MB -9.6%) vs baseline: +4.9%

✅ very_low_match

Time: ✅ 2.895ms (SLO: <8.500ms 📉 -65.9%) vs baseline: +0.8%

Memory: ✅ 70.339MB (SLO: <75.000MB -6.2%) vs baseline: +5.0%

✅ sethttpmeta - 32/32

✅ all-disabled

Time: ✅ 10.453µs (SLO: <20.000µs 📉 -47.7%) vs baseline: ~same

Memory: ✅ 32.578MB (SLO: <34.000MB -4.2%) vs baseline: +4.8%

✅ all-enabled

Time: ✅ 39.957µs (SLO: <50.000µs 📉 -20.1%) vs baseline: +0.3%

Memory: ✅ 32.519MB (SLO: <34.000MB -4.4%) vs baseline: +4.7%

✅ collectipvariant_exists

Time: ✅ 40.719µs (SLO: <50.000µs 📉 -18.6%) vs baseline: +0.2%

Memory: ✅ 32.598MB (SLO: <34.000MB -4.1%) vs baseline: +4.8%

✅ no-collectipvariant

Time: ✅ 39.936µs (SLO: <50.000µs 📉 -20.1%) vs baseline: -0.2%

Memory: ✅ 32.578MB (SLO: <34.000MB -4.2%) vs baseline: +5.0%

✅ no-useragentvariant

Time: ✅ 38.650µs (SLO: <50.000µs 📉 -22.7%) vs baseline: +0.2%

Memory: ✅ 32.558MB (SLO: <34.000MB -4.2%) vs baseline: +4.8%

✅ obfuscation-no-query

Time: ✅ 40.522µs (SLO: <50.000µs 📉 -19.0%) vs baseline: ~same

Memory: ✅ 32.637MB (SLO: <34.000MB -4.0%) vs baseline: +5.0%

✅ obfuscation-regular-case-explicit-query

Time: ✅ 75.466µs (SLO: <90.000µs 📉 -16.1%) vs baseline: -0.2%

Memory: ✅ 32.971MB (SLO: <34.000MB -3.0%) vs baseline: +4.9%

✅ obfuscation-regular-case-implicit-query

Time: ✅ 76.245µs (SLO: <90.000µs 📉 -15.3%) vs baseline: ~same

Memory: ✅ 32.952MB (SLO: <34.000MB -3.1%) vs baseline: +4.8%

✅ obfuscation-send-querystring-disabled

Time: ✅ 153.904µs (SLO: <170.000µs -9.5%) vs baseline: -0.3%

Memory: ✅ 32.952MB (SLO: <34.500MB -4.5%) vs baseline: +4.7%

✅ obfuscation-worst-case-explicit-query

Time: ✅ 148.446µs (SLO: <160.000µs -7.2%) vs baseline: ~same

Memory: ✅ 32.971MB (SLO: <34.500MB -4.4%) vs baseline: +4.7%

✅ obfuscation-worst-case-implicit-query

Time: ✅ 154.744µs (SLO: <170.000µs -9.0%) vs baseline: ~same

Memory: ✅ 32.971MB (SLO: <34.500MB -4.4%) vs baseline: +5.0%

✅ useragentvariant_exists_1

Time: ✅ 39.309µs (SLO: <50.000µs 📉 -21.4%) vs baseline: -0.6%

Memory: ✅ 32.558MB (SLO: <34.000MB -4.2%) vs baseline: +4.9%

✅ useragentvariant_exists_2

Time: ✅ 40.326µs (SLO: <50.000µs 📉 -19.3%) vs baseline: -0.5%

Memory: ✅ 32.598MB (SLO: <34.000MB -4.1%) vs baseline: +5.0%

✅ useragentvariant_exists_3

Time: ✅ 39.723µs (SLO: <50.000µs 📉 -20.6%) vs baseline: -0.6%

Memory: ✅ 32.617MB (SLO: <34.000MB -4.1%) vs baseline: +4.8%

✅ useragentvariant_not_exists_1

Time: ✅ 40.242µs (SLO: <50.000µs 📉 -19.5%) vs baseline: +1.9%

Memory: ✅ 32.598MB (SLO: <34.000MB -4.1%) vs baseline: +4.9%

✅ useragentvariant_not_exists_2

Time: ✅ 39.219µs (SLO: <50.000µs 📉 -21.6%) vs baseline: -0.1%

Memory: ✅ 32.539MB (SLO: <34.000MB -4.3%) vs baseline: +4.7%

✅ tracer - 6/6

✅ large

Time: ✅ 30.612ms (SLO: <32.950ms -7.1%) vs baseline: +0.7%

Memory: ✅ 33.286MB (SLO: <34.500MB -3.5%) vs baseline: +4.8%

✅ medium

Time: ✅ 2.964ms (SLO: <3.200ms -7.4%) vs baseline: -0.5%

Memory: ✅ 31.988MB (SLO: <34.000MB -5.9%) vs baseline: +4.5%

✅ small

Time: ✅ 336.177µs (SLO: <370.000µs -9.1%) vs baseline: -0.3%

Memory: ✅ 32.067MB (SLO: <34.000MB -5.7%) vs baseline: +4.8%

ℹ️ Scenarios Missing SLO Configuration (9 scenarios)

The following scenarios exist in candidate data but have no SLO thresholds configured:

coreapiscenario-core_dispatch_listeners
coreapiscenario-core_dispatch_no_listeners
coreapiscenario-core_dispatch_with_results_listeners
coreapiscenario-core_dispatch_with_results_no_listeners
djangosimple-baseline
errortrackingdjangosimple-baseline
errortrackingflasksqli-baseline
flasksimple-baseline
flasksqli-baseline

sabrenner

lgtm if we're good with the variable name!

ddtrace/llmobs/_llmobs.py

## Description (public change) Adds `reasoning` as an argument to `submit_evaluation_for()` and `submit_evaluation()`. This arg is used to denote an explanation behind the evaluation results (i.e. why was the span marked as toxic?) (internal change - not facing users) Also changes how the `assessment` field is stored on the evaluation object (#14792 added it as a nested `success_criteria` object) to a top-level field on the evaluation object. This isn't breaking (since this hasn't been officially released on our product backend) nor a user-facing change.  ## Testing  ## Risks  ## Additional Notes

WIP success criteria assessment for custom evals

1405fda

Add tests, release note

72faca7

Yun-Kim marked this pull request as ready for review October 14, 2025 17:21

Yun-Kim requested review from a team as code owners October 14, 2025 17:21

Yun-Kim requested review from gnufede and vitor-de-araujo October 14, 2025 17:21

sabrenner approved these changes Oct 14, 2025

View reviewed changes

gsvigruha reviewed Oct 14, 2025

View reviewed changes

ddtrace/llmobs/_llmobs.py Outdated Show resolved Hide resolved

Rename arg to assessment

8d1e5d0

Yun-Kim force-pushed the yunkim/evals-success-criteria branch from d9f1a40 to 8d1e5d0 Compare October 14, 2025 21:55

Yun-Kim enabled auto-merge (squash) October 15, 2025 15:23

Merge branch 'main' into yunkim/evals-success-criteria

b9840bb

Yun-Kim merged commit 1e8d008 into main Oct 15, 2025
549 of 552 checks passed

Yun-Kim deleted the yunkim/evals-success-criteria branch October 15, 2025 20:02

Yun-Kim mentioned this pull request Oct 16, 2025

feat(llmobs): add reasoning for custom evals #14919

Merged

feat(llmobs): success criteria assessment for custom evals #14792

feat(llmobs): success criteria assessment for custom evals #14792

Uh oh!

Conversation

Yun-Kim commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Testing

Risks

Additional Notes

Uh oh!

github-actions bot commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bootstrap import analysis

Summary

Import time breakdown

Uh oh!

pr-commenter bot commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Performance SLOs

✅ add_aspect

✅ add_inplace_aspect

✅ add_inplace_noaspect

✅ add_noaspect

✅ bytearray_aspect

✅ bytearray_extend_aspect

✅ bytearray_extend_noaspect

✅ bytearray_noaspect

✅ bytes_aspect

✅ bytes_noaspect

✅ bytesio_aspect

✅ bytesio_noaspect

✅ capitalize_aspect

✅ capitalize_noaspect

✅ casefold_aspect

✅ casefold_noaspect

✅ decode_aspect

✅ decode_noaspect

✅ encode_aspect

✅ encode_noaspect

✅ format_aspect

✅ format_map_aspect

✅ format_map_noaspect

✅ format_noaspect

✅ index_aspect

✅ index_noaspect

✅ join_aspect

✅ join_noaspect

✅ ljust_aspect

✅ ljust_noaspect

✅ lower_aspect

✅ lower_noaspect

✅ lstrip_aspect

✅ lstrip_noaspect

✅ modulo_aspect

✅ modulo_aspect_for_bytearray_bytearray

✅ modulo_aspect_for_bytes

✅ modulo_aspect_for_bytes_bytearray

✅ modulo_noaspect

✅ replace_aspect

✅ replace_noaspect

✅ repr_aspect

✅ repr_noaspect

✅ rstrip_aspect

✅ rstrip_noaspect

✅ slice_aspect

✅ slice_noaspect

✅ stringio_aspect

✅ stringio_noaspect

✅ strip_aspect

✅ strip_noaspect

✅ swapcase_aspect

✅ swapcase_noaspect

✅ title_aspect

✅ title_noaspect

✅ translate_aspect

✅ translate_noaspect

Yun-Kim commented Oct 7, 2025 •

edited

Loading

github-actions bot commented Oct 7, 2025 •

edited

Loading

github-actions bot commented Oct 7, 2025 •

edited

Loading

pr-commenter bot commented Oct 7, 2025 •

edited

Loading