Pointing + counting benchmarks (3/5) - #322
Conversation
…provider First of five PRs splitting the vision branch (#278) into milestones. This one makes any VLM checkpoint promptable with images; the benchmarks follow. Requests carry images (`LMRequest.images`) and the harness passes them through. The prompt family (`prompt_templates` / `system_prompt_style`) is a task config field settable per run, because mm_olmo derives the prompt from settings recorded in each checkpoint's own config. The HuggingFace provider gains an image-text-to-text path with transformers-5 shims for the Molmo2 remote code and fp32/bf16-autocast numerics matching mm_olmo, and loads consolidated OLMo-core exports by converting weights in-memory to the released layout. The new OLMo-core VLM provider runs raw MultimodalLM checkpoints in three on-disk formats, decoded with a batched variable-length KV cache. It lands as a package split by responsibility (checkpoint / conversion / preprocessing / cache / provider), content-identical to the reviewed #278 code by AST comparison. Also the operational fixes that running this at scale required: an env override for the 900s provider-init ceiling, an eager tqdm lock before parallel task preparation, and the missing extras mapping for the new provider kind. Co-authored-by: Ziqi Gao (Roy) <royg@allenai.org>
Add `molmo2-8b` and `molmo2-o-7b` presets mirroring the validated 4B eval settings (fp32 weights + bf16 autocast, max_crops=24 — mm_olmo evaluates the whole family with the same protocol). Collapse the provider-init timeout override to a single expression; the env var is the point, the wrapper wasn't.
143a62a to
7beea67
Compare
undfined
left a comment
undfined
left a comment
There was a problem hiding this comment.
I found four issues in this layer. The first two affect persisted result correctness and task identity, so I think they should be fixed before merge. The other two affect pretrain prompt fidelity and the documented scoring failure behavior.
| yield output.metadata["pointing_result"] | ||
|
|
||
|
|
||
| @dataclass(frozen=True) |
There was a problem hiding this comment.
These metrics will persist the wrong per-instance values. Metric.compute_instance() falls back to the scorer channel, and PointingScorer returns only F1. As a result, exact storage and pairwise analysis record F1 for precision, recall, the bucketed metrics, and the presence metrics. The point-count metric family has the same problem: close, valid, per-count accuracy, and category-average all fall back to correct. The aggregate compute() results are still right, which makes this easy to miss. Please add metric-specific compute_instance() implementations for decomposable metrics, returning None when an instance is outside the metric's bucket, and disable scorer fallback for weighted or category-level metrics that do not have an exact per-instance representation. A regression test that inspects the persisted instance metrics would catch both families.
| return build_pointing_prompt( | ||
| label, | ||
| index, | ||
| prompt_templates=self.config.prompt_templates or self.default_prompt_templates, |
There was a problem hiding this comment.
Can we include both prompt-family settings in TaskConfig.to_dict()? The generated request now depends on prompt_templates and system_prompt_style, but neither field participates in the task hash. I confirmed that uber_model_v2 / demo_or_style_v2 and none / style_and_length_v2 currently produce the same hash despite producing different prompts. That can make artifact names collide and can cause storage or pairwise tooling to treat incompatible runs as equivalent. Please serialize both fields and add a test asserting that changing either one changes the task hash.
| for idx in range(len(ds)): | ||
| ex = ds[idx] | ||
| yield Instance( | ||
| question=self.apply_family_prefix(pixmo_count_question(ex["label"], idx)), |
There was a problem hiding this comment.
This bypasses prompt_templates=none for pretrain checkpoints. The upstream formatter uses the bare lowercased label in none mode and only applies the point_count: family prefix, but this always expands the label through an instruction-style counting template. For example, the intended pretrain prompt is point_count: cats; this produces point_count: How many cats are there .... Please branch on the effective prompt_templates value here: use label.lower() for none, otherwise use pixmo_count_question(...), and then apply the family prefix. A test for both prompt families would prevent this from drifting.
| for rles in meta["pointing_annotators"]: | ||
| masks: list[np.ndarray] = [] | ||
| for seg in rles: | ||
| m = decode_segmentation(seg, image_h, image_w) |
There was a problem hiding this comment.
This does not implement the zero-score failure behavior documented by PointingTask. Exceptions from mask decoding propagate through synchronous scoring, so a missing pycocotools installation or a malformed RLE aborts the entire run rather than scoring that instance as zero. Please either catch decoding/scoring failures per output and attach a complete zero-valued pointing_result (plus useful error metadata), or make the task fail fast during initialization and update the documentation accordingly. A test with a decoding exception would make the intended contract explicit.
Per review on the split PR: - The mm_olmo unpickle shim consults sys.modules before find_spec, which raises ValueError once the shim (registered with __spec__ = None) is installed — repeat provider construction in one process crashed before. - logprobs mirrors OlmoCoreProvider's boundary handling: the limit resolves from request.max_length, an empty continuation returns an empty zero-logprob result, an over-limit continuation raises instead of failing inside gather, and the context truncates only after those checks. - The split-vocab embedding does a two-table lookup instead of caching a concatenated copy of the whole embedding matrix (~2.5 GB at the 8B size). - Package overrides for provider.kind=olmo_core_vlm normalize like olmo_core (version shorthand, git-URL binding, bundled-extra replacement), and the kind joins the sequential-only batching allowlist so a streaming config fails validation rather than serializing behind the provider lock. - The cached-decode SDPA mask construction is a module-level helper with direct tests for causal prefill, or_mask reopening and alignment, and left-padding semantics. - The unused TaskConfig prompt-family fields move out of this PR; they land with their first consumer. Tests: shim called twice, mask cases above, split-vocab lookup vs a concatenated table, and logprob boundaries (empty / at-limit / over-limit / per-request / non-positive).
Second of five PRs splitting #278. Introduces the `evals/vision/` package — tasks / scoring / benchmarks / data, per the layout proposed in review — and lands the first family in it: PixMo-Cap dense captioning, GPT-judge scored. `VisionTask` consolidates the instance caching, limit handling and image-attached request construction that every vision family previously reimplemented; families override only how instances are built and which images attach. Dense caption applies its limit inside instance building so the raw-line index driving its seeded prompt never shifts, which the base's slice then leaves untouched. The prompt-family machinery (mm_olmo's 43 seeded pointing templates, the style prefixes, and the caption templates) lives in `scoring/prompts*`; the judge in `scoring/judges`. Vision scorers are deliberately not re-exported through `common/scorers/__init__.py`: that module is imported by every task via `tasks/common/base.py`, so an eager re-export from `evals.vision` would create a package cycle. Both captioning tasks now declare `required_secrets=("OPENAI_API_KEY",)` — the judge always needed the key, but beaker never mounted it, so remote runs failed at scoring time. Registration mirrors `evals/tasks`: one `from . import vision` in `evals/__init__.py`, with the benchmark imports explicit inside the package. Co-authored-by: Ziqi Gao (Roy) <royg@allenai.org>
Third of five PRs splitting #278. Lands the pointing family (7 tasks: the question-form pixmo_points_eval / sa_co_gold_subset and the model-prompt `_mp` variants including the sampled and unsampled gold point sets) and the two counting benchmarks that share its prompt-family machinery. `PointingTask` and `ImageQATask` now subclass `VisionTask`, dropping the byte-identical instance-caching and request-construction each carried. Point-in-mask scoring (COCO-RLE decode + maximum bipartite matching, mirroring mm_olmo's SegmentationPointingScorer) lives in `vision/scoring/pointing`; the count parse ladder and the counting scorer in `count_parsing`/`vqa`, where the image-QA scorers will join them. pycocotools and scipy join the `hf` extra: the pointing scorer needs them, and a missing scorer dependency scores every instance zero rather than failing. Verified against the mm_olmo-validated runs on #278: instance counts exact for all seven datasets (1215 / 4994 / 3856 / 28672 / 166766 / 490 / 540) and first-example prompts byte-identical, including the seeded `_mp` templates. Co-authored-by: Ziqi Gao (Roy) <royg@allenai.org>
7beea67 to
70a2143
Compare
2 participants
Description
Third of five PRs splitting #278 (stack: #316 → #317 → this). Milestone: the complete pointing/counting family — 9 tasks, both suites, mm_olmo
_mpparity.Tasks
pixmo_points_eval/_mpsa_co_gold_subset/_mpsa_co_gold_point_4k_mpsa_co_gold_point_mpcountbench_qa,pixmo_countpoint_countstyle)Suites:
molmo2_pointing,molmo2_pointing_mp.Consolidation (Tyler's #5, continued)
PointingTaskandImageQATasksubclassVisionTaskand lose their byte-identicalinstances/format_requestblocks.tasks/single_image.pylands here because the counting benchmarks areImageQATasks that share the pointing prompt family; the QA benchmarks reuse it in PR 4.Scoring
scoring/pointing.py— COCO-RLE mask decode + maximum-bipartite point matching (pycocotools/scipy, lazily imported), the independent max-over-annotators, and the empty-GT⇒1.0 convention, all mirroring mm_olmo'sSegmentationPointingScorer.scoring/count_parsing.py— the vendored count parse ladder.scoring/vqa.pystarts with the shared answer helpers +PointCountScorer; the QA scorers append in PR 4.pycocotools + scipy join the
hfextra — the judge-style silent-zero failure mode (every instance failing to score reports f1 = 0.0000 with statusSuccess) is why these are declared task dependencies too.Verification
_mptemplates (pointing: <label>/ 43-template forms)pixmo_points_eval_mp0.8152 vs mm_olmo's stored 0.8083,sa_co_gold_subset_mp0.6631 vs 0.6658 — with 3,856/3,856 and 1,215/1,215 stored-prompt paritytycleanType of Change
Checklist
Part of #278 (tracking). Next: PR 4 — image QA.