Skip to content

fix(server): return capture_ui_snapshot images as ImageContent - #112

Merged
Arenukvern merged 2 commits into
Arenukvern:mainfrom
dipsy:fix/capture-ui-snapshot-image-content
Jul 31, 2026
Merged

fix(server): return capture_ui_snapshot images as ImageContent#112
Arenukvern merged 2 commits into
Arenukvern:mainfrom
dipsy:fix/capture-ui-snapshot-image-content

Conversation

@dipsy

@dipsy dipsy commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

Problem

capture_ui_snapshot is registered without an onSuccess transform (inspection_tools.dart), so runCommand falls through to the default path and agentResultToMcpResult serialises the entire bundle into one TextContent. With saveImagesToFiles off (the default), screenshots.images holds inline base64, so a single 1512×882 desktop frame ships ~145 KB of base64 as text.

Clients that cap tool output reject the result outright, which makes the tool the docs promote as step 1 of the debugging workflow ("unified evidence") unusable over MCP unless the server is started with --save-images. Observed with Claude Code: fmt_capture_ui_snapshot fails with "result (145,873 characters) exceeds maximum allowed tokens", while fmt_get_screenshots against the same app and the same VM extension returns fine.

The CLI is unaffected — fmtk exec prints the CoreResult envelope and never reaches the capability tool layer — so the documented CLI recipes keep their current output.

Change

Give capture_ui_snapshot the artifact contract get_screenshots already uses:

  • captured frames become ImageContent blocks (AgentArtifact.text(image, mimeType: 'image/png'));
  • the rest of the bundle is emitted as text JSON with screenshots.images emptied — imageSummaries still carries the per-image ids/hashes, so the JSON stays self-describing;
  • the fileUrls branch is untouched: with --save-images there are no inline images and the bundle stays a single TextContent.

Verification

  • dart test in packages/server_capability_core — 241 tests pass, including two new cases (images lifted into image/png artifacts with the JSON stripped; fileUrls bundle still a single text block).
  • End-to-end against a live Flutter macOS app through the rebuilt stdio server: tools/call fmt_capture_ui_snapshot now returns 2 content blocks — text 15,524 chars + image (image/png) — instead of one 145 KB text block.

Summary by CodeRabbit

  • New Features

    • UI snapshot results now provide bundle details separately from captured PNG images.
    • Multiple captured images are returned as individual image artifacts for easier viewing and processing.
    • Snapshot results without embedded images continue to be provided as JSON.
  • Bug Fixes

    • Preserved complete snapshot metadata while removing embedded image data from the JSON payload.

capture_ui_snapshot registered no onSuccess transform, so the whole
bundle — including the base64 frames under screenshots.images — was
serialised into a single TextContent. One 1512x882 desktop frame is
~145 KB of base64, which overflows the response budget of MCP clients
that cap tool output, making the documented "unified evidence" entry
point unusable over MCP (the CLI path is unaffected: it prints the
CoreResult envelope and never reaches this layer).

Lift the images out of the bundle into ImageContent artifacts, the same
contract get_screenshots already uses, and emit the remaining bundle
JSON as text with screenshots.images emptied. imageSummaries still
carries the per-image ids and hashes, so the JSON stays self-describing.
The fileUrls branch is untouched: with --save-images there are no inline
images and the bundle remains a single TextContent.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Jul 28, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

capture_ui_snapshot now returns bundle JSON plus separate PNG artifacts for embedded screenshots, while preserving file URL bundles as a single JSON artifact. Documentation and success-path tests cover both response shapes.

Changes

UI snapshot artifacts

Layer / File(s) Summary
Snapshot response transformation and validation
packages/server_capability_core/lib/src/tools/inspection_tools.dart, packages/server_capability_core/test/tools/inspection_tools_test.dart
capture_ui_snapshot lifts screenshots.images into image/png artifacts, clears those images from the JSON bundle, preserves file URL bundles as JSON, and tests both behaviors.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: arenukvern

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Description check ⚠️ Warning The description covers the change well, but it doesn't follow the repository template and is missing the required Summary, Validation, Checklist, and Notes sections. Reformat it to the template and add the missing sections with validation commands, checklist items, and notes.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title is concise and accurately describes the main change: returning capture_ui_snapshot images as ImageContent.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
packages/server_capability_core/lib/src/tools/inspection_tools.dart (1)

194-196: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Use explicit local types in the changed paths.

  • packages/server_capability_core/lib/src/tools/inspection_tools.dart#L194-L196: declare concrete Map<String, Object?> and List<String> types for the decoded bundle values.
  • packages/server_capability_core/test/tools/inspection_tools_test.dart#L598-L631: declare concrete types for the runner, registration result, decoded bundle, and image collections.

As per coding guidelines, “Explicit type declarations must be used.”

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/server_capability_core/lib/src/tools/inspection_tools.dart` around
lines 194 - 196, The decoded values in inspection_tools.dart at lines 194-196
must use explicit local types: declare bundle and screenshots as Map<String,
Object?> and images as List<String>. In
packages/server_capability_core/test/tools/inspection_tools_test.dart lines
598-631, add concrete type declarations for the runner, registration result,
decoded bundle, and image collections, preserving the existing behavior.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@packages/server_capability_core/lib/src/tools/inspection_tools.dart`:
- Around line 194-196: The decoded values in inspection_tools.dart at lines
194-196 must use explicit local types: declare bundle and screenshots as
Map<String, Object?> and images as List<String>. In
packages/server_capability_core/test/tools/inspection_tools_test.dart lines
598-631, add concrete type declarations for the runner, registration result,
decoded bundle, and image collections, preserving the existing behavior.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: f8152a11-3cd1-4a64-8a1e-b291573a27c5

📥 Commits

Reviewing files that changed from the base of the PR and between 6975f88 and 043dc7a.

📒 Files selected for processing (2)
  • packages/server_capability_core/lib/src/tools/inspection_tools.dart
  • packages/server_capability_core/test/tools/inspection_tools_test.dart

@Arenukvern

Copy link
Copy Markdown
Owner

hi @dipsy, thank you very much for the PR!
@all-contributors please add @dipsy code maintenance bug

@allcontributors

Copy link
Copy Markdown
Contributor

@Arenukvern

@dipsy already contributed before to code, maintenance, bug

@Arenukvern
Arenukvern merged commit 3667891 into Arenukvern:main Jul 31, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants