Chad

A small Hinglish chat model, built from scratch. Chad is a roughly 100M parameter decoder, a hand written tokenizer and a Llama style transformer, pretrained on romanized Hinglish and then fine tuned to talk like a chill Gen Z Indian kid. It runs fully in the browser, no server and no API.

Model on HuggingFace: vermarjun/chad · Build log: docs/ · In browser demo: chad/

What this is

A learning project taken end to end: tokenizer, model, data pipeline, pretraining, supervised fine tuning, preference alignment, and a real product around it. Every part is hand written and commented so it can be read top to bottom. The goal was never a state of the art model, it was to understand how a modern LLM is actually built, on free hardware, in a language that had no clean dataset to begin with.

Architecture

A modern Llama style decoder, all written by hand in src/.

Piece	Choice
Tokenizer	byte level BPE, 32k vocab, trained on the corpus
Positions	rotary (RoPE)
Attention	grouped query attention, 12 query heads, 3 kv heads
Feed forward	SwiGLU
Norm	RMSNorm, pre norm
Size	hidden 768, 12 layers, 1024 context, about 100M params

How it was built

Reddit + HF Hinglish  ->  clean + dedup  ->  4.13B token corpus  ->  pretrain (base v22)
                                                                          |
                                        distill persona data (GPT + DeepSeek teachers)
                                                                          v
                                                   SFT (v1..v4.3)  ->  DPO  ->  chad
                                                                          |
                                                 export to int8 ONNX  ->  runs in the browser

• Data: about 4.13B tokens of romanized Hinglish, 96 percent from 200 Indian subreddits, the rest from public HuggingFace sets. Full provenance and the subreddit list are in docs/data_provenance.md.
• Pretraining: on free Kaggle TPU. The base is v22, val loss about 3.77.
• SFT: the persona was taught with conversations generated by teacher LLMs, not scraped. v2 (pure distillation) is the champion.
• DPO: a final preference pass so it leans toward the sharper reply on its own.
• Inference: exported to int8 ONNX and run client side with transformers.js, around 78 tokens per second on WASM.

The full writeup is in docs/, numbered 01 through 11, from architecture to limitations.

Where the weights and data are

Everything is on HuggingFace under vermarjun:

• Model: vermarjun/chad, the base plus every SFT and DPO checkpoint and the transformers ready exports.
• Datasets: chad-pretrain, chad-sft, chad-dpo.

Versions

Version	What it is
base v22	the pretrained base
sft-v2	champion, pure distillation
sft-v3, v4, v4.2, v4.3	broader blends with tools, traded persona for coverage
dpo	the final aligned model

Repo layout

src/           the model, written by hand
train*.py      pretrain, SFT and DPO trainers
scripts/       data, distillation, sft, eval and export pipelines
notebooks/     data prep and per version inference notebooks
docs/          the full build log (01..11) plus data provenance
runs/          training run logs (weights are gitignored, on HF)
kernels/       Kaggle kernel metadata
app/           the Next.js web app and Express backend
browser-chat/  the in browser inference demo

Run it

uv venv --python 3.12
uv pip install -e ".[dev]"
.venv/bin/python -m pytest          # model and data tests

To test a trained version locally, pull a checkpoint from vermarjun/chad and open the matching notebook in notebooks/inference_*.ipynb. To pick the project back up later, start with HANDOFF.md.

Privacy

This repo is code only. Training data, model weights, and any personal chat exports stay out of git (see .gitignore). Private WhatsApp and Instagram data was parked for an experiment and never made it into any released model or dataset.