Skip to content

README.md

Latest commit

 

History

History
80 lines (60 loc) · 4.19 KB

README.md

File metadata and controls

80 lines (60 loc) · 4.19 KB
Raw

Win11React GUI Grounding Dataset (v4.2, 500 scenes)

Generated by the WebOSScrapper pipeline against win11React.

Stats

  • 54,425 records (image + bounding box + natural-language instruction)
  • 2,013 unique screenshots across 500 scenes
  • 15 apps: Explorer, Edge, Settings, Spotify, Calculator, Notepad, Whiteboard, Word, Discord, VS Code, Terminal, Mail, Camera, Microsoft Store, Task Manager
  • 6 layouts: single, corners, mosaic, cascading, split2, many_cascading
  • 4 viewports: 1920×1080, 1920×1200, 2560×1440, 1680×1050
  • ~75% accuracy (audited on 100 random samples; ~25% problem rate, dominated by sidebar mismatches and toolbar-icon offsets)

Format (ScreenSpot-Pro compatible)

Each line of screenspot.jsonl is a JSON object:

{
  "id": "win11react_scene_0042__step03_3apps_msedge_127",
  "img_filename": "outputs/dataset_v42_500/scenes/scene_0042/screenshots/step03_3apps_msedge.png",
  "instruction": "Click the close button of the Edge browser window",
  "bbox": [1872, 0, 1922, 28],
  "img_size": [1920, 1080],
  "application": "win11react",
  "platform": "web",
  "ui_type": "icon",
  "_dom_role": "button",
  "_dom_class": "uicon hvclose",
  "_apps_visible": ["MSEDGE", "WHITEBOARD", "CALCUAPP"]
}

Provenance flags

Records carry tags showing how the label was produced:

  • _relabel_v2: true — label generated by Gemini 2.5 Flash via _relabel_dropped_v2.py (recovered from records the original VLM labeling stage dropped due to API errors)
  • _qa_full_pass: true — label was reviewed by a second Gemini 2.5 Flash pass; either kept as-is or edited
  • _human_corrected: true — manually verified by a human reviewer
  • _agent_qa_edited: true — edited by a Claude vision agent in the multi-agent QA pass

Pipeline

The dataset was built in 6 stages by 16_collect_dataset.py (not included in this repository — the final harvester scripts were never committed to the source repo and must be recovered from the training cluster):

  1. Scene capture (Playwright) — open N apps in a layout, take a screenshot per intermediate state
  2. DOM bbox scan (13_full_dom_scan.py, not included) — extract every visible interactable element + its bounding box, with per-app occlusion checks
  3. Pixel filters — drop elements smaller than 14×14, blank crops, oversized container regions
  4. VLM labeling (Gemini 2.5 Flash) — sees the marked screenshot + DOM hint, produces a 4–14 word ScreenSpot-Pro-style instruction
  5. VLM QA pass (Gemini 2.5 Flash) — verifies each label by re-examining the marked screenshot; edits or drops mismatches
  6. Rule-based cleanup — drops records with named-app-on-taskbar (DOM doesn't track slot order reliably), wrong-size window-control bboxes, and quoted-text-vs-DOM mismatches

Known limitations

  • Taskbar slot misindexing — the DOM doesn't reliably link taskbar <div> slots to their app identity, so labels naming a specific app on the taskbar (e.g., "Click the Spotify icon on the taskbar") were dropped wholesale. Generic taskbar labels (Start button, search icon, system tray) are kept.
  • Window-control swaps — the maximize/restore icon (two overlapping squares) is occasionally labeled as "minimize". Most of these were caught by the bbox-size filter, but ~5–10% slip through.
  • Sidebar/icon mismatches — when a sidebar contains many similar icons, the DOM label may not match the visually-rendered one.

Files

  • screenspot.jsonl — the dataset (52 MB, 54,425 records)
  • stats.json — generation stats (scene count, layout/viewport distribution, model used, drop counts)

The screenshots themselves (~2,000 PNGs, ~3.8 GB) are not included in this branch — they are distributed on HuggingFace (Chengheng/gui-dataset-win11react-v42-500, fetch via scripts/download_dataset.py) and are regenerable from the pipeline. The original regeneration command (script not included in this repository, see note above) was:

python 16_collect_dataset.py \
    --output outputs/dataset_v42_500 \
    --num-scenes 500 \
    --vlm google/gemini-2.5-flash \
    --max-workers 12 \
    --no-llm

License

Same as parent repository.