Skip to content

fix: nested eval directory with housekeeping and result checking - #198

Open
RihanArfan wants to merge 1 commit into
mainfrom
fix/nested-eval-results
Open

fix: nested eval directory with housekeeping and result checking#198
RihanArfan wants to merge 1 commit into
mainfrom
fix/nested-eval-results

Conversation

@RihanArfan

Copy link
Copy Markdown

Fixes handling of nested eval directories (e.g. caching/cache-bypass, frameworks/h3) across result caching, housekeeping, and refingerprinting.

Eval fixtures can be organized into nested subdirectories (e.g. evals/<group>/<name>/), and saveResults() preserves this hierarchy on disk under results/<experiment>/<timestamp>/<group>/<name>/. However, several consumer paths relied on a single-level readdirSync(tsDir):

  1. Housekeeping (housekeep): A 1-level readdir encountered the parent group directory (e.g. caching/), which lacks summary.json at the group root. It evaluated the group directory as an incomplete result and recursively rmSync'd it, deleting all valid results beneath it.
  2. Result Reuse (scanReusableResults): Evaluated only 1-level directory entries against fingerprint keys, causing nested evals to never match and preventing result caching.
  3. Refingerprinting (carryForwardConfigChanges): Evaluated only 1-level directory entries, silently dropping nested evals and failing to carry forward benign config bumps.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant