Skip to content

Commit 51b69fb

Browse files
authored
fix(cua): nest python gemini screenshots in FunctionResponse, drop unused openai dep (#157)
## Summary Two bugbot findings on commit \`c684ca7\`: 1. **Medium** — Python Gemini provider sent screenshots as a separate \`Part(inline_data=...)\` entry in the user content after the \`FunctionResponse\` part. With multiple function calls per turn the model can't bind a screenshot to its originating call. The standalone \`python/gemini-computer-use\` template and the TS unified template both nest the screenshot as a \`FunctionResponsePart\` inside \`FunctionResponse.parts\`. This PR matches that structure and adds the predefined-actions allowlist that gates screenshot inclusion. 2. **Low** — \`openai\` was listed in \`pyproject.toml\` but never imported. The OpenAI provider uses raw \`httpx\` against the Responses API. Removed. ## Test plan - [ ] Smoke run python cua with Gemini against a multi-call turn and confirm screenshot binds to the originating call - [ ] \`uv sync\` after dep change <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Medium Risk** > Moderate risk because it changes the structure of Gemini tool-call response parts, which could affect how multi-call turns are interpreted by the model or SDK. Dependency removal is low risk but may impact downstream installs if they relied on the extra package. > > **Overview** > **Gemini Python CUA now nests screenshots inside each tool call response.** Instead of sending a standalone `Part(inline_data=...)` after the `FunctionResponse`, screenshots are attached as `FunctionResponse.parts` (as `FunctionResponsePart`/`FunctionResponseBlob`) so multi-call turns can reliably associate images with the correct action; screenshot inclusion is gated by a `PREDEFINED_ACTIONS` allowlist. > > **Template deps cleanup.** Removes the unused `openai` dependency from `pyproject.toml`. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit ee48a5c. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY -->
1 parent c684ca7 commit 51b69fb

2 files changed

Lines changed: 23 additions & 10 deletions

File tree

pkg/templates/python/cua/providers/gemini.py

Lines changed: 23 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,9 @@
1616
Tool,
1717
ComputerUse,
1818
Environment,
19+
FunctionResponse,
20+
FunctionResponseBlob,
21+
FunctionResponsePart,
1922
)
2023

2124
from . import CuaProvider, TaskOptions, TaskResult
@@ -29,6 +32,13 @@
2932
PX_PER_NOTCH = 60
3033
MAX_NOTCHES_PER_ACTION = 17
3134

35+
PREDEFINED_ACTIONS = {
36+
"click_at", "hover_at", "type_text_at", "scroll_document",
37+
"scroll_at", "wait_5_seconds", "go_back", "go_forward",
38+
"search", "navigate", "key_combination", "drag_and_drop",
39+
"open_web_browser",
40+
}
41+
3242
def _system_prompt() -> str:
3343
date = datetime.now().strftime("%A, %B %d, %Y")
3444
return (
@@ -115,20 +125,24 @@ async def run_task(self, options: TaskOptions) -> TaskResult:
115125
)
116126

117127
if result.get("error"):
118-
responses.append(Part.from_function_response(
128+
responses.append(Part(function_response=FunctionResponse(
119129
name=fc.name,
120130
response={"error": result["error"], "url": "about:blank"},
121-
))
131+
)))
122132
else:
123-
responses.append(Part.from_function_response(
133+
fr_parts = None
134+
if result.get("screenshot") and fc.name in PREDEFINED_ACTIONS:
135+
fr_parts = [FunctionResponsePart(
136+
inline_data=FunctionResponseBlob(
137+
mime_type="image/png",
138+
data=result["screenshot"],
139+
),
140+
)]
141+
responses.append(Part(function_response=FunctionResponse(
124142
name=fc.name,
125143
response={"url": result.get("url", "about:blank")},
126-
))
127-
if result.get("screenshot"):
128-
responses.append(Part(inline_data={
129-
"mime_type": "image/png",
130-
"data": result["screenshot"],
131-
}))
144+
parts=fr_parts,
145+
)))
132146

133147
contents.append(Content(role="user", parts=responses))
134148

pkg/templates/python/cua/pyproject.toml

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,5 @@ dependencies = [
99
"google-genai>=1.71.0",
1010
"httpx>=0.28.1",
1111
"kernel>=0.47.0",
12-
"openai>=2.30.0",
1312
"python-dotenv>=1.2.2",
1413
]

0 commit comments

Comments
 (0)