Summary
agent-eval discovers nested fixtures correctly, but housekeeping treats the first path segment as a complete eval result directory and can delete all nested fixture results as "incomplete".
Nested fixture names are supported by discovery, for example:
evals/901-create-component-atom-reshaped/concise/PROMPT.md
evals/901-create-component-atom-reshaped/detailed/PROMPT.md
evals/901-create-component-atom-reshaped/explicit-stories/PROMPT.md
discoverFixtures() returns names like:
901-create-component-atom-reshaped/concise
901-create-component-atom-reshaped/detailed
901-create-component-atom-reshaped/explicit-stories
Results are saved with matching nested paths:
results/cc/<timestamp>/901-create-component-atom-reshaped/concise/summary.json
results/cc/<timestamp>/901-create-component-atom-reshaped/detailed/summary.json
results/cc/<timestamp>/901-create-component-atom-reshaped/explicit-stories/summary.json
But after the experiment completes, housekeeping scans only the immediate children of results/<experiment>/<timestamp>/. It sees:
results/cc/<timestamp>/901-create-component-atom-reshaped/
as if it were one eval result directory, checks for summary.json directly inside it, marks it incomplete, and removes it recursively. That deletes all nested variant results.
Observed Behavior
After a run with nested fixtures, the terminal printed:
Housekeeping: removed 0 duplicate(s), 1 incomplete, 0 non-model failure(s)
The saved nested result directories were present while the experiment was running, then disappeared when housekeeping ran.
Expected Behavior
Housekeeping should preserve complete nested fixture results. Either:
- recurse to actual eval result directories containing
summary.json, or
- use the known fixture names / fingerprints rather than treating immediate timestamp children as eval result directories.
Source Pointers
discoverFixtures() explicitly supports nested names like vercel-cli/deploy.saveResults() writes eval results under join(experimentDir, evalSummary.name), preserving slashes in nested fixture names.housekeep() reads only immediate timestamp children and calls isComplete() on those directories, so the parent segment of a nested fixture is incorrectly classified as incomplete.
Workaround
Flatten fixture names, for example:
901-create-component-atom-reshaped-concise
901-create-component-atom-reshaped-detailed
901-create-component-atom-reshaped-explicit-stories
Summary
agent-evaldiscovers nested fixtures correctly, but housekeeping treats the first path segment as a complete eval result directory and can delete all nested fixture results as "incomplete".Nested fixture names are supported by discovery, for example:
discoverFixtures()returns names like:Results are saved with matching nested paths:
But after the experiment completes, housekeeping scans only the immediate children of
results/<experiment>/<timestamp>/. It sees:as if it were one eval result directory, checks for
summary.jsondirectly inside it, marks it incomplete, and removes it recursively. That deletes all nested variant results.Observed Behavior
After a run with nested fixtures, the terminal printed:
The saved nested result directories were present while the experiment was running, then disappeared when housekeeping ran.
Expected Behavior
Housekeeping should preserve complete nested fixture results. Either:
summary.json, orSource Pointers
discoverFixtures()explicitly supports nested names likevercel-cli/deploy.saveResults()writes eval results underjoin(experimentDir, evalSummary.name), preserving slashes in nested fixture names.housekeep()reads only immediate timestamp children and callsisComplete()on those directories, so the parent segment of a nested fixture is incorrectly classified as incomplete.Workaround
Flatten fixture names, for example: