Background
Now skill-up can run evals, generate reports, and create baseline benchmark data. But I think it still misses one offline command to compare two finished runs.
When we change a Skill, review a PR, or upgrade model, we often want to know:
- which cases are fixed
- which cases are regressed
- how pass rate changes
- how tokens and duration change
- whether engine or model is different
- whether this can be used as a CI gate
Proposal
Add a new command:
skill-up compare <old-result.json> <new-result.json>
It should compare two existing result/report JSON files. It should not run eval again, and should not call any agent or judge.
Scope
The command can:
- read JSON files compatible with current report.Input/result.json format
- use primary case result semantics, so benchmark without_skill entries are not counted twice
- show run level delta:
- pass rate
- case counts
- total/input/output tokens
- duration
- show case level transition:
- fixed
- regressed
- unchanged
- added
- removed
- show metadata difference, like skill name, engine, model, schema version and time
- support default text output
- support --format json for CI or other tools
- support optional gates, for example fail on regression or too much token increase
Non-goals
- Do not rerun evals
- Do not call agent or judge
- Do not add dashboard/cloud/snapshot feature in this issue
- First version does not need markdown output
Acceptance criteria
skill-up compare old.json new.json prints a readable summary
--format json prints stable machine readable output
- benchmark result is not double counted
- fixed/regressed/added/removed cases are classified correctly
- configured gate failure returns non-zero exit code
- invalid input gives clear error message
Background
Now skill-up can run evals, generate reports, and create baseline benchmark data. But I think it still misses one offline command to compare two finished runs.
When we change a Skill, review a PR, or upgrade model, we often want to know:
Proposal
Add a new command:
It should compare two existing result/report JSON files. It should not run eval again, and should not call any agent or judge.
Scope
The command can:
Non-goals
Acceptance criteria
skill-up compare old.json new.jsonprints a readable summary--format jsonprints stable machine readable output