Skip to content

Add meta-research loop to program.md#417

Open
PodJamz wants to merge 1 commit intokarpathy:masterfrom
PodJamz:meta-research-loop
Open

Add meta-research loop to program.md#417
PodJamz wants to merge 1 commit intokarpathy:masterfrom
PodJamz:meta-research-loop

Conversation

@PodJamz
Copy link
Copy Markdown

@PodJamz PodJamz commented Mar 25, 2026

Summary

Addresses #314. Adds a ## Meta-Research section to program.md that instructs the agent to periodically review its own experiment history and refine its research strategy.

What this adds

Every 20 experiments, the agent:

  1. Scans results.tsv for repeated failure patterns (same hypothesis type discarded 3+ times)
  2. Appends confirmed dead-ends to hypotheses_blacklist.md (untracked, like results.tsv)
  3. Re-evaluates which research phases are producing gains vs stalling
  4. Adjusts its active research strategy accordingly

What this does NOT change

  • No Python files modified
  • The fixed outer evaluation loop (val_bpb, 5-minute budget) is never touched
  • The agent can only modify its research strategy, never its evaluation criteria

Why

The current loop optimizes train.py but the research strategy in program.md is static. After 50+ experiments, the agent may keep trying hypothesis classes that have been proven unproductive. A meta-research step lets the agent learn from its own experiment history and focus on productive directions.

This is inspired by the HyperAgent framework (Zhang, Lehman, Clune - Meta FAIR, 2025) - the principle that the improvement mechanism itself should be subject to improvement. The outer eval loop stays fixed; the research strategy evolves.

Addresses karpathy#314. Instructs the agent to periodically review its own
experiment history, identify repeated failure patterns, maintain a
hypotheses blacklist, and refine its research strategy.

Inspired by the HyperAgent framework (Meta FAIR, 2025) - the concept
that the improvement mechanism itself should be improvable.
@PodJamz PodJamz force-pushed the meta-research-loop branch from dbdecb9 to 4411c4c Compare March 25, 2026 21:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant