WinDOM: Self-Family Distillation for Small-Model GUI Grounding

Chengheng Li Chen, Zhiqian Zhou, Hao Chen, Nicolas Chauvin — ICML 2026 submission

Links: Paper (arXiv) · Project website · WinDOM corpus (HF) · Cleaned split (HF)

We study how far a small (2B) GUI grounding model can be pushed without any external teacher. We introduce WinDOM, a 54,425-record GUI grounding corpus harvested by driving an open-source Windows 11 web reimplementation (win11react) under headless Playwright and reading ground-truth bounding boxes directly off the DOM. On top of it we propose Self-Family Distillation (SFD) — rejection-sampling SFT from a frozen same-family 4B teacher prompted with a noisy GT-bbox hint (SFD-4B), or from an EMA copy of the student itself (self-EMA) — followed by GRPO, where the saturation depth of the cold-start checkpoint is an explicit hyperparameter: under-saturated inits (step 100) beat converged ones (step 900). SFD-4B + early-init GRPO improves over base Qwen3.5-2B by +5.4 OOD-mean accuracy (+3.5 ScreenSpot-Pro, +7.0 OSWorld-G, +5.8 ScreenSpot-V2); the fully self-referential self-EMA variant nearly matches the 4B teacher (65.2 vs 66.3 OOD-mean).

The paper itself is in paper/ (windom_paper.pdf); the corpus is documented in data/README.md.

Repository layout

WinDOM/
├── README.md                  ← this file
├── pyproject.toml / requirements.txt
├── run_from_config.py         ← scraper pipeline entry point (config-driven)
├── paper/                     ← main.tex, main.bib, windom_paper.pdf, styles,
│   └── figures/               ← make_figures.py + shipped figure PDFs + assets/
├── launchers/                 ← 13 run_train_v42_*.sh (training recipes + the
│                                checkpoint-init saturation ablation grid) +
│                                3 template_eval_*.sh (eval wrappers)
├── src/
│   ├── webos_scrapper/        ← WebOSScrapper package: predecessor trajectory
│   │                            pipeline (crawl → annotate → dedup → explore;
│   │                            Strategy pattern in trajectory/runner.py)
│   └── windom/                ← main package: training + evaluation
│       ├── cli.py             ← Facade: `python -m windom <command>` dispatches
│       │                        to the module mains (lazy heavy imports)
│       ├── core/              ← shared lib (prompts / parsing / scoring / data /
│       │                        misc)
│       ├── training/          ← sft.py, sfd_4b.py, sfd_self_ema.py, grpo.py
│       └── evaluation/        ← own_test.py, screenspot.py, osworld_g.py,
│                                test_official_prompt.py (sanity suite)
├── runners/                   ← run_win11_scrapper.py (Playwright CSS-selector
│                                trajectory executor), launch_parallel.py
├── configs/                   ← config_win11react.yml (pipeline config)
├── trajectories/              ← win11_css_500.jsonl (500 CSS-selector trajectories)
├── tests/                     ← test_strategies.py (unit tests, no browser/LLM needed)
├── data/
│   ├── README.md              ← WinDOM corpus documentation (format, pipeline, stats)
│   └── stats.json             ← generation stats
└── scripts/
    ├── download_dataset.py        ← fetches the WinDOM corpus from HuggingFace
    └── download_eval_benchmarks.py← fetches ScreenSpot-V2 / ScreenSpot-Pro

Installation

python3 -m venv .venv && source .venv/bin/activate
pip install -e .            # both packages (windom + webos_scrapper) + the `windom` CLI
pip install -r requirements.txt

# Scraper only (dataset pipeline): browser backend
playwright install chromium

requirements.txt covers both the scraper (playwright, openai, pyyaml, tqdm) and the training/eval stack (torch, transformers, trl, datasets, accelerate, huggingface_hub, Pillow; wandb is optional — all launchers pass --wandb). Install the torch build matching your CUDA version. Training and evaluation are single-GPU workloads.

Reproducing the paper

All launchers resolve the repository root from their own location, export PYTHONPATH=$REPO_ROOT/src, and invoke the unified CLI (python -m windom <command> — same as the windom console script from pip install -e .), so they can be run from anywhere. Export HF_TOKEN (and WANDB_API_KEY for training), or put them in ~/.env. Every command also works directly, e.g. python -m windom train-sft --help.

1. Download the data

# WinDOM corpus: raw screenshots + cleaned training split (~54K records)
python3 scripts/download_dataset.py
#   → data/win11react_v42_500/  (incl. gemma4_ready/{train,val,test}.jsonl)

# ScreenSpot-V2 / ScreenSpot-Pro eval benchmarks
python3 scripts/download_eval_benchmarks.py
#   → eval_benchmarks/{screenspot_v2,screenspot_pro}/samples.json

OSWorld-G is streamed directly from HuggingFace (MMInstruction/OSWorld-G) by the eval script — nothing to download.

2. SFT baseline (cold-start arm + baseline rows)

bash launchers/run_train_v42_sft.sh

Produces $CKPT_ROOT/v42_sft/checkpoint-{100..1400} (default ./checkpoints/), pushed to Chengheng/webos-v42-sft.

3. SFD cold-starts

bash launchers/run_train_v42_4b_hint.sh    # SFD-4B: frozen 4B hint teacher
bash launchers/run_train_v42_self_ema.sh   # self-EMA: teacherless variant

Produces checkpoint-{100..1400} every 100 steps, pushed to Chengheng/webos-v42-4b-hint / Chengheng/webos-v42-self-ema (run_train_v42_self_ema_4b.sh is the 4B student=teacher arm).

4. GRPO with saturation-depth choice

Champion recipe (SFD-4B cold-start, under-saturated cp100 init):

bash launchers/run_train_v42_grpo_4bh.sh

The full ablation grid — same command, different --model/--subfolder init:

Init checkpoint	Launcher
SFT cp100 / cp900	run_train_v42_grpo.sh / run_train_v42_grpo_sft_cp900.sh
SFD-4B cp100 / cp900	run_train_v42_grpo_4bh.sh / run_train_v42_grpo_4bhint_cp900.sh
Self-EMA cp100 / cp900	run_train_v42_grpo_ema_cp100.sh / run_train_v42_grpo_ema_cp900.sh
Self-EMA-4B cp100	run_train_v42_grpo_emacp100_4b.sh
Seed replicates (error bars)	run_train_v42_grpo_4bh_seed123.sh, run_train_v42_grpo_4bh_seed2024.sh

Each run saves checkpoint-{50,100,...} under $CKPT_ROOT and pushes to its --hf-repo. RL hyperparameters (num_generations=8, lr 2e-6, β=0, dapo loss, temperature 1.3) are in the launchers.

5. Evaluate

# Held-out WinDOM test set
CHECKPOINT=Chengheng/webos-v42-grpo-4bhint-cp900 SUBFOLDER=checkpoint-550 \
    bash launchers/template_eval_own_test.sh

# ScreenSpot-V2 (BENCHMARK=screenspot_pro for ScreenSpot-Pro)
CHECKPOINT=Chengheng/webos-v42-4b-hint SUBFOLDER=checkpoint-200 \
    BENCHMARK=screenspot_v2 bash launchers/template_eval_screenspot.sh

# OSWorld-G
CHECKPOINT=Chengheng/webos-v42-grpo-4bhint-cp900 SUBFOLDER=checkpoint-100 \
    bash launchers/template_eval_osworld_g.sh

All evals use greedy decoding, enable_thinking=False, and point-in-bbox click accuracy; each writes eval_results/<tag>/results.jsonl + eval.log. Expected metric keys: click_in_gt (own test / ScreenSpot) and hit (OSWorld-G). Published checkpoints (Chengheng/webos-v42-*) let you skip steps 2–4 entirely.

6. Figures and paper PDF

cd paper/figures && python3 make_figures.py   # reads eval_results/, writes fig*.pdf
cd ../.. && cd paper && pdflatex main && bibtex main && pdflatex main && pdflatex main

Sanity checks (no GPU needed)

PYTHONPATH=src python3 -m unittest discover -s tests   # scraper unit tests
PYTHONPATH=src python3 src/windom/evaluation/test_official_prompt.py
                                                       # prompt/parse/score tests
                                                       # (needs training deps installed)

Corpus construction

The WinDOM corpus was built in 6 stages (Playwright scene capture → DOM bbox scan → pixel filters → VLM labeling → VLM QA → rule-based cleanup), documented in data/README.md. Note: the final harvester scripts were never committed to the source repository and are not included here (they must be recovered from the training cluster); the corpus itself is fully available on HuggingFace via step 1, and src/webos_scrapper/ + runners/ document the predecessor trajectory pipeline the harvester grew out of.

Citation

@article{chen2026windom,
  title   = {WinDOM: Self-Family Distillation for Small-Model GUI Grounding},
  author  = {Chen, Chengheng Li and Zhou, Zhiqian and Chen, Hao and Chauvin, Nicolas},
  journal = {arXiv preprint arXiv:2606.25964},
  year    = {2026}
}

Data & model artifacts

• Corpus: Chengheng/gui-dataset-win11react-v42-500 (raw), Chengheng/webos-grounding-clean-v2 (cleaned)
• Checkpoints: Chengheng/webos-v42-{sft,4b-hint,self-ema,self-ema-4b} (cold-starts), Chengheng/webos-v42-grpo-* (RL runs)
• Benchmarks: OS-Copilot/ScreenSpot-v2, likaixin/ScreenSpot-Pro, MMInstruction/OSWorld-G