kloop — Loop Engineering for Data Science (v0.2)

A reusable harness for shipping data-science projects using a long-running multi-agent loop with six loop families running a 4-stage cycle.

Built on:

• Andrej Karpathy's LOOPS.md (paper/LOOPS.md) — 9 rules for letting the model drive
• Sairam's "20 Loop Design Patterns" (X article, 2026-07-01) — six-loop taxonomy around Generate → Evaluate → Learn → Improve
• Anthropic's multi-agent patterns — orchestrator-workers, evaluator-optimizer

"A prompt is a thing you type once and forget. A loop is a thing that runs while you sleep." — Karpathy, LOOPS.md

The v0.2 architecture

                         ORCHESTRATOR  (no model — pure I/O)
                    reads memory-state.json, dispatches stages
                    ┌──────────────────────────────────────────┐
                    │                                          │
   ┌────────────┐   │   ┌───────────┐    ┌──────────┐          │
   │  GENERATE  │──▶│──▶│ EVALUATE  │───▶│  LEARN   │──┐       │
   │  builder   │   │   │ grader    │    │ meta-    │  │       │
   │  writes    │   │   │ writes    │    │ critic   │  │       │
   │  src/,     │   │   │ eval/<i>/ │    │ writes   │  │       │
   │  numbers   │   │   │ report.md │    │ lessons  │  │       │
   └────────────┘   │   └───────────┘    └──────────┘  │       │
                    │                                     │       │
                    │   ┌───────────┐                     │       │
                    │◀──│ IMPROVE   │◀────────────────────┘       │
                    │   │ strategist│                             │
                    │   │ decides   │                             │
                    │   │ next iter │                             │
                    │   └─────┬─────┘                             │
                    │         │                                   │
                    │   ┌─────▼─────┐                             │
                    │   │ MEMORY    │  ← schema-enforced state   │
                    │   │ archivist │     resume in <30 sec       │
                    │   └───────────┘                             │
                    └──────────────────────────────────────────┘
                              │
                              ▼
              SIX LOOP FAMILIES  (each runs its own micro-loop)
              ┌──────────┬──────────┬──────────┬──────────┬──────────┬──────────┐
              │ PLANNING │ QUALITY  │ MEMORY   │EXPLORATION│OPTIMIZ'N │ LEARNING │
              │strategist│ grader   │archivist │researcher │ tuner    │meta-crit │
              └──────────┴──────────┴──────────┴──────────┴──────────┴──────────┘

What's new in v0.2

• Six loop families instead of three roles: planning, quality, memory, exploration, optimization, learning
• Schema-enforced state (memory-state.json with JSON Schema 2020-12)
• Lessons family extracts insights per iteration and recommends contract patches
• Optimization family runs budget-capped sweeps
• Orchestrator script drives the cycle, persists state, decides iterate vs restart
• Drift signals embedded in memory-state for automatic restart detection

Repo layout (v0.2)

kloop/
├── README.md
├── LICENSE
├── .gitignore
├── paper/
│   ├── LOOPS.md                ← Karpathy's paper, verbatim reformatting
│   └── RULES-INDEX.md          ← 9 rules mapped to DS
├── skill/
│   ├── SKILL.md                ← umbrella doc, start here
│   ├── loops/                  ← six loop families
│   │   ├── planning/           strategist: sprint + contract
│   │   ├── quality/            grader: report + anchors
│   │   ├── memory/             archivist: schema-enforced state
│   │   ├── exploration/        researcher: feature_list + hypotheses
│   │   ├── optimization/       tuner: sweeps with budgets
│   │   └── learning/           meta-critic: lessons + patches
│   ├── references/             ← cross-cutting
│   ├── scripts/
│   │   ├── kloop               ← CLI dispatcher
│   │   ├── orchestrator.py     ← multi-agent driver (state machine)
│   │   └── tdd_helpers.py      ← leakage / drift / calibration checks
│   ├── templates/              ← v0.1 templates (kept for back-compat)
│   └── examples/               ← canonical churn example
└── examples/
    └── churn-loop/             ← worked example, all six families in action

Quickstart

Bootstrap a new project

git clone https://github.com/PraveenKumarSridhar/kloop.git
cd kloop
./skill/scripts/kloop init my-ds-project
cd my-ds-project
ls .kloop/   # memory-state.json, sprint.md, contract.md, feature_list.json, hypothesis.md, lessons.md

Run the orchestrator

# One full cycle
python3 skill/scripts/orchestrator.py run

# Or one stage at a time
python3 skill/scripts/orchestrator.py run --stage generate
python3 skill/scripts/orchestrator.py run --stage evaluate
python3 skill/scripts/orchestrator.py run --stage learn
python3 skill/scripts/orchestrator.py run --stage improve

# Status and validation
python3 skill/scripts/orchestrator.py status
python3 skill/scripts/orchestrator.py validate

The orchestrator:

1. Reads .kloop/memory-state.json to decide which stage is next.
2. Dispatches the appropriate specialist (Generate / Evaluate / Learn / Improve) as a hermes delegate_task subagent with the stage-specific system prompt.
3. Persists state after each stage with schema validation.
4. Decides iterate (loop back to Generate) or restart (archive state, fresh from contract) based on memory-state signals.

Study the example

cd examples/churn-loop
python3 data/generate_synthetic.py
python3 src/train.py --seed 7
make eval
python3 ../../skill/scripts/orchestrator.py status

The 9 Karpathy rules, mapped to v0.2

#	LOOPS.md rule	v0.2 grounding
I	Write the loop, not the prompt	The 4-stage cycle is the procedure. Six families are the loops.
II	Separate the roles	Six contexts, six prompts, six artifact namespaces. Hard isolation.
III	Negotiate the contract first	Planning family negotiates with Quality family. Contract frozen before Generate runs.
IV	Write to disk, not to context	memory-state.json is the single source of truth. Schema-enforced.
V	Let the loop restart	Improve stage detects restart condition; orchestrator archives and resets.
VI	Score the subjective	Quality family scores 4 DS axes (design/correctness/craft/communication) with anchor projects.
VII	Read the traces	Learning family reads eval/<iter>/train.log, numbers.json, lessons.md.
VIII	Delete the harness	kloop audit flags skills/loops// files deletable after model upgrades.
IX	The bottleneck always moves	Improve stage explicitly picks the next bottleneck to attack.

DS-specific additions

These are the design choices that make kloop DS-correct, not just agent-correct:

1. Held-out evaluation set is the load-bearing artifact. Path + SHA-256 pinned in contract.md. Generate-loop is forbidden from importing it (lint-enforced).
2. Leakage assertions are first-class contract items. Target ↔ future correlation, permutation importance, shuffle-label AUC, no transitive holdout imports.
3. Feature ledger (feature_list.json). Every feature tried, with status (tried/kept/killed/leaked). Prevents re-litigating failed features across iterations.
4. Four DS subjective axes (design / correctness / craft / communication) with 3 good + 3 slop anchor projects per axis.
5. Latency, calibration, drift as standard contract items. Drift PSI between train and holdout must be < 0.20.
6. Sweeps have budgets. Optimization-loop has a per-sweep compute/time cap (max trials OR max wall-time, whichever first).
7. memory-state.json schema includes drift_signals[]. The Memory family watches for drift between train and current state; if drift exceeds threshold, it flags the contract for review.

Provenance

• Karpathy, A. (2026). LOOPS.md: Field Notes on Agents That Run for Days. Verbatim reformatting at paper/LOOPS.md. Permission: "Personal use of this material is permitted... Freely available; ideas subject to revision as the models change."
• Sairam, R. (2026-07-01). 20 Loop Design Patterns Every AI Engineer Should Know. X article id 2072234929048649728. Six-loop taxonomy (planning, quality, memory, exploration, optimization, learning) around the Generate→Evaluate→Learn→Improve cycle.
• Anthropic (2024-2026). Building Effective Agents. Multi-agent architectural patterns (orchestrator-workers, evaluator-optimizer) underpin the Generate/Evaluate separation.
• This repo: Original work by Praveen Kumar Sridhar, MIT-licensed.

Iteration

Things I'd like to add next:

• Real delegate_task wiring in the orchestrator (currently prints what would be invoked).
• A second worked example in a different domain (forecasting or retrieval).
• Populated assets/rubric/ with real reference good/slop projects per axis.
• Hash-chained history[] in memory-state for tamper detection.
• A kloop doctor command that runs all 32 failure-mode checks (see references/failure-modes.md) and reports.

Open an issue or PR.

— pk, July 2026