To enable rigorous evaluation and empower researchers to develop their own agents, we built the <strong><a href="https://github.com/cocoabench/cocoa-agent">CocoaAgent</a></strong> framework. CocoaAgent provides seamless integration with <a href="https://github.com/agent-infra/sandbox">AIO Sandbox</a>, an all-in-one Docker environment. It equips agents with a full suite of tools—browser automation, terminal access, file operations, and code interpreters—enabling them to operate like human developers in realistic settings. Our framework is model-agnostic, and we provide example scripts for running agents with both open-source LLMs such as Qwen3-VL and commercial models such as GPT-5.1. To support robust evaluation at scale, CocoaAgent implements both dynamic runtime tests for verifying computational correctness and lightweight static-matching checks for deterministic answers. We are currently producing comprehensive results on CocoaBench using this framework, which will be available soon on our leaderboard.
0 commit comments