Skip to content

HANDOFF.md

Latest commit

 

History

History
49 lines (32 loc) · 3.56 KB

HANDOFF.md

File metadata and controls

49 lines (32 loc) · 3.56 KB
Raw

Chad handoff

Everything you need to pick this project back up. Chad is a small from scratch Hinglish chat model (about 100M params): a hand built byte level tokenizer and a Llama style decoder (RoPE, grouped query attention, SwiGLU, RMSNorm), pretrained on romanized Hinglish, fine tuned with distilled persona data, then DPO aligned.

Where everything lives now

All weights and datasets are backed up on HuggingFace (private, account vermarjun):

  • Model vermarjun/chad — base, all SFT and DPO checkpoints, transformers exports, side experiments.
  • Datasets vermarjun/chad-sft, vermarjun/chad-dpo, vermarjun/chad-pretrain.

Kept locally for testing: every best.pt inference weight in the *_logs/out folders, the transformers exports in export/, all training logs (log.csv, progress.txt), and the private data.

Deleted locally to save space, all recoverable: the raw reddit dumps, the cleaned corpus, the packed bins, and the big intermediates. See docs/data_provenance.md to rebuild them.

The version map

The base is pretrained once. Everything else fine tunes from it.

Version What it is Local HF
base v22 pretrained base, val loss about 3.77 runs/pretrain/v22_logs/out/best.pt chad/base/v22.pt
sft-v2 champion, pure distillation runs/sft/sft_base_to_v2_logs/out/best.pt chad/checkpoints/sft-v2.pt, chad/transformers/v2
sft-v1 early reddit mining, too wholesome, dropped runs/sft/sft_v1_logs/out/best.pt chad/checkpoints/sft-v1.pt
sft-v3 v2 plus general Hinglish, voice diluted runs/sft/sft_base_to_v3_logs/out/best.pt chad/checkpoints/sft-v3.pt
sft-v4 / v4.2 / v4.3 distill heavy blends with calc and gif tags, over roast sft_base_to_v4*, runs/sft/sft_v4_final/out/best.pt chad/checkpoints/sft-v4*, chad/transformers/v4*
dpo-final preference tuned on top, latest stage runs/dpo/dpo_final/out/best.pt chad/checkpoints/dpo-final.pt, chad/transformers/dpo

Full eval detail in docs/07-evaluation/.

How to test a version locally

The inference notebooks load best.pt directly: notebooks/inference_*.ipynb, one per version. The browser path uses the transformers exports in export/, converted to onnx, then run in browser-chat/. Export pipeline is scripts/export.

How it was built (pointers)

The docs/ folder has the full story in 8 numbered sections: architecture, tokenizer, data collection, sft and distillation, training, compute, evaluation. Code: src/ is the model package, train.py and train_tpu.py are the trainers, scripts/ holds the data, distill, sft, eval and export pipelines.

Key learnings (so you do not repeat them)

  • Pretraining ran on free Kaggle TPU. v18 to v20 crashed (weight tying blew up the param count, SDPA fp32 OOM, host OOM). v22 froze for about 20 minutes at every checkpoint until the save interval was raised, it was dirty page writeback from a 1.2GB checkpoint.
  • The 100M param size is the real ceiling on humor. Data changes format, persona and length, but not how funny the model is. Stop chasing funny with more data.
  • v2 (pure persona distillation) beat the broader v3 and v4 blends, which went split personality.
  • HuggingFace upload_large_folder hangs on the xet path for big folders. Upload large datasets with per shard upload_file instead.

To continue

  • Rebuild the data: docs/data_provenance.md plus docs/reddit_subreddits.txt.
  • Retrain: warm start from base v22, see docs/05-training/.
  • The teacher engines for distillation were gpt-5.4-mini and DeepSeek v4-flash, both non thinking.