Multi-image benchmarks + multimodal docs (5/5) - #324
Open
jason718 wants to merge 7 commits into
Open
Conversation
…provider First of five PRs splitting the vision branch (#278) into milestones. This one makes any VLM checkpoint promptable with images; the benchmarks follow. Requests carry images (`LMRequest.images`) and the harness passes them through. The prompt family (`prompt_templates` / `system_prompt_style`) is a task config field settable per run, because mm_olmo derives the prompt from settings recorded in each checkpoint's own config. The HuggingFace provider gains an image-text-to-text path with transformers-5 shims for the Molmo2 remote code and fp32/bf16-autocast numerics matching mm_olmo, and loads consolidated OLMo-core exports by converting weights in-memory to the released layout. The new OLMo-core VLM provider runs raw MultimodalLM checkpoints in three on-disk formats, decoded with a batched variable-length KV cache. It lands as a package split by responsibility (checkpoint / conversion / preprocessing / cache / provider), content-identical to the reviewed #278 code by AST comparison. Also the operational fixes that running this at scale required: an env override for the 900s provider-init ceiling, an eager tqdm lock before parallel task preparation, and the missing extras mapping for the new provider kind. Co-authored-by: Ziqi Gao (Roy) <royg@allenai.org>
Add `molmo2-8b` and `molmo2-o-7b` presets mirroring the validated 4B eval settings (fp32 weights + bf16 autocast, max_crops=24 — mm_olmo evaluates the whole family with the same protocol). Collapse the provider-init timeout override to a single expression; the env var is the point, the wrapper wasn't.
Per review on the split PR: - The mm_olmo unpickle shim consults sys.modules before find_spec, which raises ValueError once the shim (registered with __spec__ = None) is installed — repeat provider construction in one process crashed before. - logprobs mirrors OlmoCoreProvider's boundary handling: the limit resolves from request.max_length, an empty continuation returns an empty zero-logprob result, an over-limit continuation raises instead of failing inside gather, and the context truncates only after those checks. - The split-vocab embedding does a two-table lookup instead of caching a concatenated copy of the whole embedding matrix (~2.5 GB at the 8B size). - Package overrides for provider.kind=olmo_core_vlm normalize like olmo_core (version shorthand, git-URL binding, bundled-extra replacement), and the kind joins the sequential-only batching allowlist so a streaming config fails validation rather than serializing behind the provider lock. - The cached-decode SDPA mask construction is a module-level helper with direct tests for causal prefill, or_mask reopening and alignment, and left-padding semantics. - The unused TaskConfig prompt-family fields move out of this PR; they land with their first consumer. Tests: shim called twice, mask cases above, split-vocab lookup vs a concatenated table, and logprob boundaries (empty / at-limit / over-limit / per-request / non-positive).
Second of five PRs splitting #278. Introduces the `evals/vision/` package — tasks / scoring / benchmarks / data, per the layout proposed in review — and lands the first family in it: PixMo-Cap dense captioning, GPT-judge scored. `VisionTask` consolidates the instance caching, limit handling and image-attached request construction that every vision family previously reimplemented; families override only how instances are built and which images attach. Dense caption applies its limit inside instance building so the raw-line index driving its seeded prompt never shifts, which the base's slice then leaves untouched. The prompt-family machinery (mm_olmo's 43 seeded pointing templates, the style prefixes, and the caption templates) lives in `scoring/prompts*`; the judge in `scoring/judges`. Vision scorers are deliberately not re-exported through `common/scorers/__init__.py`: that module is imported by every task via `tasks/common/base.py`, so an eager re-export from `evals.vision` would create a package cycle. Both captioning tasks now declare `required_secrets=("OPENAI_API_KEY",)` — the judge always needed the key, but beaker never mounted it, so remote runs failed at scoring time. Registration mirrors `evals/tasks`: one `from . import vision` in `evals/__init__.py`, with the benchmark imports explicit inside the package. Co-authored-by: Ziqi Gao (Roy) <royg@allenai.org>
Third of five PRs splitting #278. Lands the pointing family (7 tasks: the question-form pixmo_points_eval / sa_co_gold_subset and the model-prompt `_mp` variants including the sampled and unsampled gold point sets) and the two counting benchmarks that share its prompt-family machinery. `PointingTask` and `ImageQATask` now subclass `VisionTask`, dropping the byte-identical instance-caching and request-construction each carried. Point-in-mask scoring (COCO-RLE decode + maximum bipartite matching, mirroring mm_olmo's SegmentationPointingScorer) lives in `vision/scoring/pointing`; the count parse ladder and the counting scorer in `count_parsing`/`vqa`, where the image-QA scorers will join them. pycocotools and scipy join the `hf` extra: the pointing scorer needs them, and a missing scorer dependency scores every instance zero rather than failing. Verified against the mm_olmo-validated runs on #278: instance counts exact for all seven datasets (1215 / 4994 / 3856 / 28672 / 166766 / 490 / 540) and first-example prompts byte-identical, including the seeded `_mp` templates. Co-authored-by: Ziqi Gao (Roy) <royg@allenai.org>
Fourth of five PRs splitting #278. Lands the flagship image-QA family — 12 benchmarks and the `molmo2_imageqa` suite (14 names with the counting pair), plus `molmo2_imageqa_caption` now spanning QA + dense caption. The scoring layer grows the vendored parsers (VQA normalization, the MMMU multiple-choice/open parser shared with mmmu_pro and the multi-image family, MathVista offline extraction, the byte-exact CharXiv grading prompts) and the QA scorer classes join the counting scorer in `scoring/vqa`. The CharXiv GPT judge and MathVista's GPT extraction join the dense-caption judge in `scoring/judges`, sharing its cached-call machinery. The judge-scored tasks (charxiv x2, math_vista) declare `required_secrets=("OPENAI_API_KEY",)` so beaker mounts the key their judges always needed. CharXiv's vendored prompts keep their trailing whitespace (it is part of the official prompt bytes); lint exempts vendored modules file-by-file. Every moved symbol is AST-verified against the reviewed #278 code; the only changed definitions are the VisionTask consolidation and the added secrets. Co-authored-by: Ziqi Gao (Roy) <royg@allenai.org>
Last of five PRs splitting #278. MuirBench, MMIU and BLINK, scored by the mm_olmo MuirBenchEval-family protocol (option-letter accuracy over an image list), and the README's Multimodal Evaluation section updated for the `evals/vision` layout. `MultiImageQATask` completes the VisionTask consolidation: it overrides only `_attach_images` (an image list capped at mm_olmo's eval-time 20) rather than carrying its own request construction. The list-image lazy loaders join the single-image ones in `vision/data/images`. Co-authored-by: Ziqi Gao (Roy) <royg@allenai.org>
jason718
force-pushed
the
jasonr/vision-split-5-multiimage-docs
branch
from
August 28, 2026 17:48
aa30039 to
3c26121
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Last of five PRs splitting #278 (stack: #316 → #317 → #322 → #323 → this). Milestone: multi-image family complete — all 25 vision tasks and 5 suites from the vision branch are on
main.Benchmarks
muir_bench (2,600), mmiu (11,698), blink (1,901) — each instance carries an image list, scored by mm_olmo's MuirBenchEval-family protocol (option-letter accuracy, with MMIU's per-category metrics). Suite:
molmo2_multiimage.Framework completion
MultiImageQATaskfinishes the VisionTask consolidation (Tyler's Codify the storage mechanics for experiment results #5): it overrides only_attach_images(image list capped at mm_olmo's eval-timemax_images=20); instance caching/limits/request construction come from the basevision/data/images.py; the multi-image MC parser+scorer merge intoscoring/multi_image.pyMOLMO_DATA_DIR,OPENAI_API_KEY, run commands, both providers, task authoring) updated for theevals/visionlayoutVerification
MultiImageQATaskconsolidation; across the whole stack, 183/183 symbols accounted fortycleanOnce this merges: #278 gets a closing summary linking the five PRs, and the follow-up issues from Tyler's review (#2 ResponseBatchScorer — which also fixes charxiv's dead 5-per-call judge batching; #3b provider-class decomposition; #4 RuntimeRequirements; #6 ImageReference laziness) get filed.
Type of Change
Checklist
Part of #278 (tracking).