ci: add Vekil performance regression checks

[sozercan]

Problem

Vekil has performance-sensitive request paths and documented Go benchmarks, but the required CI workflow does not run performance checks. A change can increase request latency, bytes per operation, or allocations per operation without failing CI.

Absolute throughput is not a reliable pull request gate on shared runners. Process benchmark results vary with runner capacity and background load, even when the code is unchanged.

Proposed approach

Add a focused Vekil performance regression job:

• Run a small set of stable, local hot-path benchmarks on pull requests.
• Compare the merge base and pull request head on the same runner with the same Go version, fixed GOMAXPROCS, fixtures, and benchmark arguments.
• Collect at least ten samples and use benchstat for timing comparisons.
• Fail only when a timing regression exceeds 10% and is statistically significant.
• Add explicit allocation ceilings for selected hot paths. Any ceiling increase should require an intentional code and test change.
• Upload raw benchmark output and the comparison report as workflow artifacts.
• Keep the job bounded so it does not materially extend normal CI time.

Run process-level saturation and fixed-rate throughput checks on a schedule or a dedicated runner. Report those as trends until the runner is stable enough for a blocking relative gate. Discard runs when a direct local control shows excessive variance. Do not gate on a fixed requests-per-second value.

Suggested initial coverage:

• Native OpenAI Chat passthrough handler
• Request-body lifecycle binding
• Canonical Chat request and response inspection
• Provider request construction and route dispatch

Acceptance criteria

• Pull requests run a required, Vekil-only performance regression job.
• The job compares base and head under matched settings.
• Statistically significant timing regressions above 10% fail the job.
• Allocation budget regressions fail the job.
• Raw results and the comparison report are retained as artifacts.
• The benchmark command and local reproduction steps are documented.
• Scheduled process benchmarks do not use an absolute throughput threshold.

[sozercan]

Orka Issue Status

State: blocked
Approval: not required

Latest update

The required pull-request CI currently runs correctness, race, build, vet, and operational checks but no benchmarks or allocation budgets, so regressions in local request hot paths cannot fail CI. The repository already documents matched ten-sample benchstat comparisons, and deterministic in-memory route construction/dispatch benchmarks are immediately usable. Focused benchmarks are still missing for the public native OpenAI Chat handler, pooled request-body lifecycle binding, and canonical request/response inspection. Add a bounded CI job that checks out the pull-request merge base and head, uses the same pinned Go and benchstat versions, fixed GOMAXPROCS and benchmark arguments, collects at least ten preferably interleaved samples, retains raw output, and fails only when benchstat reports a statistically significant time/op increase greater than 10%. Enforce separately checked-in B/op and allocs/op ceilings against head results. New benchmark names will not exist in the merge base on the initial rollout, so either stage the focused benchmarks before enabling their relative timing gate or deliberately apply one shared benchmark harness to both revisions; existing Chat route benchmarks can be gated immediately. Process-level saturation work should remain a separate scheduled trend workflow with a local variance control and no absolute requests-per-second threshold.

Blocked reason: needs_human