Image-QA benchmarks (4/5) - #323
Open
jason718 wants to merge 6 commits into
Open
Conversation
…provider First of five PRs splitting the vision branch (#278) into milestones. This one makes any VLM checkpoint promptable with images; the benchmarks follow. Requests carry images (`LMRequest.images`) and the harness passes them through. The prompt family (`prompt_templates` / `system_prompt_style`) is a task config field settable per run, because mm_olmo derives the prompt from settings recorded in each checkpoint's own config. The HuggingFace provider gains an image-text-to-text path with transformers-5 shims for the Molmo2 remote code and fp32/bf16-autocast numerics matching mm_olmo, and loads consolidated OLMo-core exports by converting weights in-memory to the released layout. The new OLMo-core VLM provider runs raw MultimodalLM checkpoints in three on-disk formats, decoded with a batched variable-length KV cache. It lands as a package split by responsibility (checkpoint / conversion / preprocessing / cache / provider), content-identical to the reviewed #278 code by AST comparison. Also the operational fixes that running this at scale required: an env override for the 900s provider-init ceiling, an eager tqdm lock before parallel task preparation, and the missing extras mapping for the new provider kind. Co-authored-by: Ziqi Gao (Roy) <royg@allenai.org>
Add `molmo2-8b` and `molmo2-o-7b` presets mirroring the validated 4B eval settings (fp32 weights + bf16 autocast, max_crops=24 — mm_olmo evaluates the whole family with the same protocol). Collapse the provider-init timeout override to a single expression; the env var is the point, the wrapper wasn't.
This was referenced Aug 25, 2026
Per review on the split PR: - The mm_olmo unpickle shim consults sys.modules before find_spec, which raises ValueError once the shim (registered with __spec__ = None) is installed — repeat provider construction in one process crashed before. - logprobs mirrors OlmoCoreProvider's boundary handling: the limit resolves from request.max_length, an empty continuation returns an empty zero-logprob result, an over-limit continuation raises instead of failing inside gather, and the context truncates only after those checks. - The split-vocab embedding does a two-table lookup instead of caching a concatenated copy of the whole embedding matrix (~2.5 GB at the 8B size). - Package overrides for provider.kind=olmo_core_vlm normalize like olmo_core (version shorthand, git-URL binding, bundled-extra replacement), and the kind joins the sequential-only batching allowlist so a streaming config fails validation rather than serializing behind the provider lock. - The cached-decode SDPA mask construction is a module-level helper with direct tests for causal prefill, or_mask reopening and alignment, and left-padding semantics. - The unused TaskConfig prompt-family fields move out of this PR; they land with their first consumer. Tests: shim called twice, mask cases above, split-vocab lookup vs a concatenated table, and logprob boundaries (empty / at-limit / over-limit / per-request / non-positive).
Second of five PRs splitting #278. Introduces the `evals/vision/` package — tasks / scoring / benchmarks / data, per the layout proposed in review — and lands the first family in it: PixMo-Cap dense captioning, GPT-judge scored. `VisionTask` consolidates the instance caching, limit handling and image-attached request construction that every vision family previously reimplemented; families override only how instances are built and which images attach. Dense caption applies its limit inside instance building so the raw-line index driving its seeded prompt never shifts, which the base's slice then leaves untouched. The prompt-family machinery (mm_olmo's 43 seeded pointing templates, the style prefixes, and the caption templates) lives in `scoring/prompts*`; the judge in `scoring/judges`. Vision scorers are deliberately not re-exported through `common/scorers/__init__.py`: that module is imported by every task via `tasks/common/base.py`, so an eager re-export from `evals.vision` would create a package cycle. Both captioning tasks now declare `required_secrets=("OPENAI_API_KEY",)` — the judge always needed the key, but beaker never mounted it, so remote runs failed at scoring time. Registration mirrors `evals/tasks`: one `from . import vision` in `evals/__init__.py`, with the benchmark imports explicit inside the package. Co-authored-by: Ziqi Gao (Roy) <royg@allenai.org>
Third of five PRs splitting #278. Lands the pointing family (7 tasks: the question-form pixmo_points_eval / sa_co_gold_subset and the model-prompt `_mp` variants including the sampled and unsampled gold point sets) and the two counting benchmarks that share its prompt-family machinery. `PointingTask` and `ImageQATask` now subclass `VisionTask`, dropping the byte-identical instance-caching and request-construction each carried. Point-in-mask scoring (COCO-RLE decode + maximum bipartite matching, mirroring mm_olmo's SegmentationPointingScorer) lives in `vision/scoring/pointing`; the count parse ladder and the counting scorer in `count_parsing`/`vqa`, where the image-QA scorers will join them. pycocotools and scipy join the `hf` extra: the pointing scorer needs them, and a missing scorer dependency scores every instance zero rather than failing. Verified against the mm_olmo-validated runs on #278: instance counts exact for all seven datasets (1215 / 4994 / 3856 / 28672 / 166766 / 490 / 540) and first-example prompts byte-identical, including the seeded `_mp` templates. Co-authored-by: Ziqi Gao (Roy) <royg@allenai.org>
Fourth of five PRs splitting #278. Lands the flagship image-QA family — 12 benchmarks and the `molmo2_imageqa` suite (14 names with the counting pair), plus `molmo2_imageqa_caption` now spanning QA + dense caption. The scoring layer grows the vendored parsers (VQA normalization, the MMMU multiple-choice/open parser shared with mmmu_pro and the multi-image family, MathVista offline extraction, the byte-exact CharXiv grading prompts) and the QA scorer classes join the counting scorer in `scoring/vqa`. The CharXiv GPT judge and MathVista's GPT extraction join the dense-caption judge in `scoring/judges`, sharing its cached-call machinery. The judge-scored tasks (charxiv x2, math_vista) declare `required_secrets=("OPENAI_API_KEY",)` so beaker mounts the key their judges always needed. CharXiv's vendored prompts keep their trailing whitespace (it is part of the official prompt bytes); lint exempts vendored modules file-by-file. Every moved symbol is AST-verified against the reviewed #278 code; the only changed definitions are the VisionTask consolidation and the added secrets. Co-authored-by: Ziqi Gao (Roy) <royg@allenai.org>
jason718
force-pushed
the
jasonr/vision-split-4-image-qa
branch
from
August 28, 2026 17:48
bab61eb to
cad212e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fourth of five PRs splitting #278 (stack: #316 → #317 → #322 → this). Milestone: the flagship
molmo2_imageqasuite — 14 tasks — runs end to end.Benchmarks (12 new; the suite's other 2 are PR 3's counting tasks)
ai2d, chart_qa, doc_qa, info_qa, math_vista, mmmu, mmmu_pro, real_world_qa, text_vqa, vqa2, charxiv_descriptive, charxiv_reasoning — with their variants (
math_vista:offline|gpt,ai2d:transparent|test,chart_qa:test).mmmu_prostays decoupled frommmmu(own prompts/parser/scorer, faithful to the official repo; primary = SOTA-reported overall).Suites:
molmo2_imageqa(AVERAGE, all 0-1 primaries) andmolmo2_imageqa_caption(DISPLAY_ONLY — dense caption reports 0-100).Scoring layer
scoring/vqa_normalization.py,multiple_choice.py(the MMMU parser, shared with mmmu_pro and the PR-5 multi-image family),math_vista_offline.py,mmmu_pro.py,charxiv.py— vendored parsers/prompts, behavior-preserving; CharXiv's trailing whitespace is part of the official prompt bytes, so lint exempts vendored modules file-by-filescoring/vqa.py; the CharXiv judge and MathVista GPT extraction join the dense-caption judge inscoring/judges.py, sharing its cached-call machineryscoring/__init__.pyre-exports the parser functions (internal toevals.vision, so no cycle withcommon/scorers)Fixes
required_secrets=("OPENAI_API_KEY",)on charxiv_descriptive/reasoning and math_vista — same beaker live-bug as dense caption in Vision task framework + captioning benchmarks (2/5) #317Verification
test_image_qa_requests.pytycleanuv run pytest tests/evals/vision/ -q # 185 passed — the whole vision family uv run pytest tests/ --ignore=tests/integration -qType of Change
Checklist
Part of #278 (tracking). Next: PR 5 — multi-image + docs.