Commit 51b69fb
authored
fix(cua): nest python gemini screenshots in FunctionResponse, drop unused openai dep (#157)
## Summary
Two bugbot findings on commit \`c684ca7\`:
1. **Medium** — Python Gemini provider sent screenshots as a separate
\`Part(inline_data=...)\` entry in the user content after the
\`FunctionResponse\` part. With multiple function calls per turn the
model can't bind a screenshot to its originating call. The standalone
\`python/gemini-computer-use\` template and the TS unified template both
nest the screenshot as a \`FunctionResponsePart\` inside
\`FunctionResponse.parts\`. This PR matches that structure and adds the
predefined-actions allowlist that gates screenshot inclusion.
2. **Low** — \`openai\` was listed in \`pyproject.toml\` but never
imported. The OpenAI provider uses raw \`httpx\` against the Responses
API. Removed.
## Test plan
- [ ] Smoke run python cua with Gemini against a multi-call turn and
confirm screenshot binds to the originating call
- [ ] \`uv sync\` after dep change
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> **Medium Risk**
> Moderate risk because it changes the structure of Gemini tool-call
response parts, which could affect how multi-call turns are interpreted
by the model or SDK. Dependency removal is low risk but may impact
downstream installs if they relied on the extra package.
>
> **Overview**
> **Gemini Python CUA now nests screenshots inside each tool call
response.** Instead of sending a standalone `Part(inline_data=...)`
after the `FunctionResponse`, screenshots are attached as
`FunctionResponse.parts` (as
`FunctionResponsePart`/`FunctionResponseBlob`) so multi-call turns can
reliably associate images with the correct action; screenshot inclusion
is gated by a `PREDEFINED_ACTIONS` allowlist.
>
> **Template deps cleanup.** Removes the unused `openai` dependency from
`pyproject.toml`.
>
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
ee48a5c. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->1 parent c684ca7 commit 51b69fb
2 files changed
Lines changed: 23 additions & 10 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
19 | 22 | | |
20 | 23 | | |
21 | 24 | | |
| |||
29 | 32 | | |
30 | 33 | | |
31 | 34 | | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
32 | 42 | | |
33 | 43 | | |
34 | 44 | | |
| |||
115 | 125 | | |
116 | 126 | | |
117 | 127 | | |
118 | | - | |
| 128 | + | |
119 | 129 | | |
120 | 130 | | |
121 | | - | |
| 131 | + | |
122 | 132 | | |
123 | | - | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
124 | 142 | | |
125 | 143 | | |
126 | | - | |
127 | | - | |
128 | | - | |
129 | | - | |
130 | | - | |
131 | | - | |
| 144 | + | |
| 145 | + | |
132 | 146 | | |
133 | 147 | | |
134 | 148 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
9 | 9 | | |
10 | 10 | | |
11 | 11 | | |
12 | | - | |
13 | 12 | | |
14 | 13 | | |
0 commit comments