[Epic] Port evalpipeline validation and artifact quality gates into Gym

[sephmard]

Background

Teams running production evals on top of Gym have built an external validation pipeline — evalpipeline — to fill a gap: Gym collects rollouts and computes aggregate metrics but has no mechanism to verify rollout quality before those metrics are reported. evalpipeline runs as a post-processing layer on top of Gym output, classifying trajectories as clean or dirty, gating aggregation on data quality, and selecting minimum clean repeats per problem before scores are surfaced.

This is a bandaid on the absence of first-class quality validation in Gym. This epic ports those capabilities natively into Gym so teams don't need an external pipeline to get trustworthy eval results.

** Internal Resources:**

• Repo: https://gitlab-master.nvidia.com/dl/JoC/competitive_evaluation/nvidia-eval-factory-benchmarking/-/tree/main/evalpipeline
• Architecture: https://gitlab-master.nvidia.com/dl/JoC/competitive_evaluation/nvidia-eval-factory-benchmarking/-/blob/08468fe245d6e31b4fcc2a25042c2eb95a77fc8e/evalpipeline/docs/architecture.md
• Example pipeline run: https://gitlab-master.nvidia.com/dl/JoC/competitive_evaluation/nvidia-eval-factory-benchmarking/-/pipelines/54441505

What evalpipeline does today

evalpipeline is a thin CLI that runs four ordered stages after a Gym eval job:

1. pre-run-check — dry-run and API endpoint probes before the eval starts; fails fast if the environment isn't ready
2. run-evaluation — runs the backend eval command with explicit overrides; records the exact command executed
3. verify-artifacts — reads raw trajectories, classifies each as clean or dirty based on: zero-token turns, missing agent steps, missed metrics, fully-zero rollouts; writes trajectories_atif.jsonl and a verification_summary.json
4. scavenge-artifacts — from the verified trajectories, selects the minimum required number of clean repeats per problem; records shortfalls (problems without enough clean repeats) in scavenge_summary.json; writes trajectories_scavenged.jsonl

The verification logic is pure and backend-independent. The architecture already anticipates Gym as a first-class backend.

What needs to be built in Gym

Rollout quality verification
After rollout collection, classify each rollout against cleanliness criteria: zero-token turns, missing agent steps, missed metrics, infrastructure placeholder responses. Surface clean/dirty counts alongside aggregate metrics. Today none of this is visible to the user.

Minimum clean repeat gating
Before reporting aggregate metrics for a benchmark, verify that each task has the required number of clean rollouts. If shortfalls exist, report which tasks are incomplete rather than silently computing metrics over partial data.

Pre-run checks
Before starting a long eval run, validate that the environment is reachable and configured correctly. Today config mistakes surface only after Ray init and server startup.

Artifact quality summary
Alongside _aggregate_metrics.json, produce a structured quality summary covering clean/dirty rollout counts, shortfalls by task, and any cleanliness failures — so users and downstream tooling can gate on data quality, not just metric values.

Out of scope

• Porting the ATIF trajectory format as a required schema for all Gym rollouts — quality verification should work against Gym's existing output format
• Replacing evalpipeline as a standalone tool for teams that need it outside Gym
• Changes to verifier or agent authoring contracts

Acceptance criteria

• Rollout quality verification runs as part of gym eval run and gym eval aggregate
• Clean/dirty classification is reported alongside aggregate metrics
• Minimum clean repeat gating is configurable per benchmark
• Tasks with insufficient clean repeats are reported as shortfalls, not silently included in aggregation
• Pre-run checks validate environment reachability before rollout collection starts
• A quality summary artifact is written alongside _aggregate_metrics.json