feat(so101_curobo): default cuRobo/URDF path to the auto-downloaded SO-101 cache URDF#165
Conversation
…ed SO-101 cache URDF
The default MuJoCo demo already auto-downloads the SO-101 model into the
external ~/.strands_robots cache (via strands_robots.assets), and that
download ships a URDF (so101_new_calib.urdf) + an assets/ mesh dir
alongside the MJCF. But the cuRobo planner and the URDF-sim path ignored
it and forced the user to hand-supply --curobo-urdf / SO101_URDF even
though a perfectly good URDF was already on disk -- a usability papercut
(the demo "ships no assets" yet still made you stage a URDF for cuRobo).
Add shared resolvers in planner.py with the documented precedence:
explicit arg -> SO101_URDF / SO101_ASSET env -> the auto-resolved cache
URDF + mesh dir (resolve_model_dir("so101"), which triggers the same
auto-download the MuJoCo path uses). Wire them into:
- CuroboMotionPlanner.__init__ (urdf_path / asset_path),
- _curobo_usable() so `make_planner(prefer="auto")` now selects cuRobo
out-of-the-box when it's installed + the cache URDF exists (previously it
fell back to scripted unless a URDF flag was set),
- controller.py's sim-arm URDF resolution (so the sim loads the EXACT URDF
cuRobo plans with),
- plan_curobo_offline.py + app.py CLI help text.
Returns None when none of the three resolve, so callers keep their existing
fail-with-hint behaviour on a box with neither a flag nor the cache. No
change to the default MuJoCo+scripted path.
Adds two unit tests (precedence + None-when-unavailable) to smoke_test.py;
the existing MuJoCo smoke + strands-labs#143 re-run tests still pass. black / isort /
flake8 clean.
cagataycali
left a comment
There was a problem hiding this comment.
Approve. This removes a real usability papercut - the example ships no assets of its own yet forced a --curobo-urdf/SO101_URDF flag even though the MuJoCo path already auto-downloads a perfectly good URDF into the cache.
The resolve_so101_urdf / resolve_so101_asset precedence (explicit kwarg -> env -> cache) is well-factored, the fallback _so101_cache_urdf degrades safely (returns (None, "") on any import/resolve failure so callers keep their fail-with-hint behaviour), and wiring controller.py through the same resolver guarantees the sim arm loads the EXACT URDF cuRobo plans against - that shared-resolver invariant is the part that matters most for correctness.
One behavioural note worth flagging for the record (not blocking): _curobo_usable now routes through resolve_so101_urdf, so on a cache miss make_planner(prefer="auto") can trigger resolve_model_dir("so101")'s auto-download as a side effect of planner selection. That is the intended out-of-the-box behaviour per the PR description and is the same download the MuJoCo path already performs, so it is acceptable for an example - just calling it out so future readers know selection can touch the network on a clean box.
Tests properly mock _so101_cache_urdf (no real network in CI) and pin all three precedence levels plus the None fallback. Lint clean, default MuJoCo+scripted path unchanged. Good to merge.
* feat: add LIBERO-on-MuJoCo baseline example (R5, #12)
Stage 2 of the umbrella (#8). Adds the first row of the backend
comparison matrix — a minimal LIBERO eval against the upstream MuJoCo
backend shipped by 'strands-robots'. Imports only from 'strands_robots',
not 'strands_robots_sim'; this repo just hosts the file so all four
'libero_*.py' siblings (mujoco / isaac / newton / newton_fleet) live
next to each other for the R15 matrix story.
Two flavours per the design discussion on the issue:
- examples/libero_mujoco.py (direct-API, ~78 LOC, deterministic)
Runs 'sim.evaluate_benchmark(...)' with policy_provider='mock' against
the first registered LIBERO spatial task. Prints two grep-stable lines
('benchmark_name=...' and 'success_rate=... wall_time=...s') so R15
can subprocess-and-parse the output for the comparison table. R15
ingests this file; R4's README placeholder gets the wall-time number
filled in from a run of this script.
- examples/libero_mujoco_agent.py (Strands-Agent, natural-language, ~90 LOC)
Headline pedagogy demo of why 'Simulation' being an 'AgentTool' buys
you anything: an LLM is given the same task in plain English, picks
the right action sequence (create_world → add_robot → evaluate_benchmark
→ destroy), and summarises. Output is non-deterministic by design;
R15 does not ingest this file. Requires a configured LLM provider
(Bedrock by default). Includes a commented appendix showing the
deterministic 'agent.tool.libero_sim(action=...)' dispatch path for
CI / scripted runs without an LLM.
- examples/README.md (~83 LOC)
Backend matrix table (mujoco row filled with measured wall-time;
isaac / newton / newton_fleet rows TBD with links to R8 / R12).
Two-flavours section explaining when to pick which file. Install
instructions including a temporary git-install line for the
'sim-mujoco' / 'benchmark-libero' extras (they're on
'strands-labs/robots' main only until the next PyPI release) and a
pointer to the upstream BDDL-parser PR (strands-labs/robots#147)
needed for 'load_libero_suite' to actually register tasks.
Measurement (single-CPU dev machine, mock policy):
benchmark_name=libero-spatial-pick_up_the_black_bowl_between_...
success_rate=0.00 wall_time=0.8s
Mock policy can't satisfy real goals so success_rate=0 is expected and
correct. The number to remember is the 0.8 s wall-time, which fills in
the 'mujoco | 1 | …' row in R4's README backend matrix once both PRs
land.
Notes / dependencies:
- Blocked-on-during-implementation: strands-labs/robots#147 (case-
insensitive BDDL parsing). Without that fix 'load_libero_suite'
registers 0 tasks and this example raises before timing anything. The
PR was filed today; once it merges the example runs end-to-end as
shown above. The wall-time reported here was measured against a local
build of that fix.
- R4 README placeholder fill-in (mujoco | 1 | <seconds>) is intentionally
NOT touched in this PR — R4 (#11 / PR #25) is still open. Once R4
merges, a small follow-up commit on this branch will swap 'TBD' for
'0.8 s'. Otherwise we'd race the merge.
- R15 (the flagship matrix, #22) will subprocess-and-parse the two
print() lines from libero_mujoco.py — keep that output format stable.
Closes #12 once the upstream parser fix and the R4 README placeholder
fill-in both land.
* docs: fill in mujoco wall-time in README backend matrix (R5 follow-up)
R4's README (PR #25) shipped with placeholder cells for every row of the
backend comparison matrix. Now that R4 has merged, swap the mujoco row's
'TBD (R5)' for the measured ~0.8 s value from PR #26's libero_mujoco.py
run on a single-CPU dev box.
Also tightens the column header to match what we actually measure
instead of the aspirational '50 eps, libero-spatial-pick_up_the_red_cube'
that was in R4 — that exact task name doesn't exist in real LIBERO and
the example uses 10 eps + the first-registered-task pattern. Header now
reads 'first registered LIBERO spatial task, 10 episodes, mock policy',
matching what R5 / R8 / R12 ship and what R15 will aggregate.
Footnote pins the dependency on the still-open upstream parser fix
strands-labs/robots#147 so readers know the task list is currently empty
without it.
Final size: 152 lines / 7.88 KB — under R4's '≤ 200 lines / ≤ 8 KB'
DoD.
* docs(readme): tick Stage 1 boxes, expand LIBERO matrix to 5 rows, even-handed backend bullets
Three small README updates following Stage 1 completion (#9 #10 #11):
1. Mark Stage 1 status checkboxes as done — R2 / R3 / R4 are merged.
2. Rewrite the "Pick X for Y" bullets so Isaac and Newton are framed
even-handedly. Both support 4096+ envs; the differentiator is
rendering vs. solver flexibility, not parallelism. Previous wording
implied Newton owned fleet-scale RL exclusively, which isn't accurate.
3. Replace the LIBERO matrix table with a 5-row version that includes
the missing `isaac | 4096` row (tracked as R23 / #27 — IsaacLab-style
fleet example) and adds Renderer / Why use this row / Example / Issue
columns so each row answers the "why pick this row" question. Drop
the redundant per-backend bullet list above the table.
Refs #8 (umbrella). Adds tracking link to #27 (R23, Isaac fleet example).
* feat: align R5 with updated #12 spec — two patterns × two policies + MP4
Re-scope per #12's rewrite:
- Two execution patterns (one-shot vs iterative) instead of two
drivers (direct-API vs natural-language). The agent-driver demo
is dropped in favour of an iterative-supervision demo that
replaces the deleted SteppedSimEnv.
- Two policy choices on every file: --policy {mock,groot}. mock
for CI / no-GPU smoke; groot for real LIBERO runs against the
public nvidia/GR00T-N1.7-LIBERO checkpoint.
- MP4 recording around every run, written to rollouts/YYYY_MM_DD/
with the deleted SimEnv's filename convention preserved (now
also encoding policy=mock|groot).
Files touched:
- examples/libero_mujoco.py — REWRITE. argparse-driven; handles both
policies; wraps evaluate_benchmark in start_cameras_recording /
stop_cameras_recording for an MP4-per-run; documents the GR00T
service-start path (Strands gr00t_inference tool + bare-Docker
fallback) in the docstring; uses data_config='libero_panda' (the
registered key — NOT the bare 'libero' the legacy SimEnv used);
prints the same two grep-stable lines R15 will subprocess-and-parse,
now with policy= and videos= fields appended.
- examples/libero_mujoco_stepped.py — NEW. Runnable instance of the
iterative supervision pattern. Same --policy flag interface as the
one-shot file. Demonstrates start_policy → time.sleep(1/OBSERVE_HZ)
→ get_state → render → (system-2 hook) loop, then stop_policy +
stop_cameras_recording. Documents the upstream API gaps it works
around (state envelope vs flat dict, render returning PNG-in-dict
vs ndarray, missing reset_benchmark public method).
- examples/libero_mujoco_agent.py — DELETED. The agent-driver flow it
demoed is replaced by the iterative pattern in the new spec.
- examples/README.md — REWRITE. New 'Two execution patterns' +
'Two policy choices' sections; mujoco wall-time row split into
groot-authoritative number (TBD until measured) plus a separate
mock-smoke-test reference paragraph; usage commands cover all four
combinations (one-shot/stepped × mock/groot).
- examples/MIGRATION.md — UPDATED. Mapping table cells now link the
runnable example files (SimEnv → libero_mujoco.py, SteppedSimEnv
→ libero_mujoco_stepped.py); side-by-side and iterative snippets
fixed to match the actual upstream API (policy_config dict instead
of policy_port, add_robot before evaluate_benchmark, status-envelope
semantics on get_state/render); new MP4 section covers the
preserved rollouts/ filename convention and the per-episode
segmentation upstream gap.
- README.md — mujoco backend-matrix row format changed to
'wall-time @ success-rate' to match the updated spec; footnote
notes the canonical number comes from --policy=groot, mock smoke
is reference-only.
Verified locally:
python examples/libero_mujoco.py --policy mock --n-episodes 10 --seed 42
benchmark_name=libero-spatial-pick_up_the_black_bowl_between_...
policy=mock success_rate=0.00 wall_time=0.8s videos=rollouts/...
python examples/libero_mujoco_stepped.py --policy mock --max-iters 5
policy=mock observe_iters=5 videos=rollouts/...
(Both ran against the upstream BDDL fix from
strands-labs/robots#147; MP4 itself was skipped by the recorder
because this dev box has no X11 display, but on a real machine the
filename pattern resolves as printed.)
Upstream gaps documented in the example docstrings + this PR
description, all small enough to file as follow-ups rather than
gate R5:
1. evaluate_benchmark has no record_video= / video_dir= kwargs —
we wrap with start_cameras_recording externally, getting one
MP4 per RUN (not per episode). Per-episode segmentation needs
upstream plumbing.
2. No public reset_benchmark(benchmark_name, seed) — the iterative
demo skips per-task scene loading and runs against the default
Panda scene; the supervision-loop pattern is identical regardless.
3. get_state() returns the standard status envelope, not a flat
{'reward': ...} dict; loop exit is on --max-iters, not reward.
4. render() returns a status dict with PNG bytes inside, not a
numpy.ndarray; helper _extract_frame_ndarray unpacks if needed.
5. data_config='libero' alias is not in policies/groot's registry;
the LIBERO Panda key is 'libero_panda'.
Builds on R5 follow-up be662a8 + the consolidation of #28 (a4a21ef),
both already on this branch.
* feat: re-pivot R5 per #12 v3 — programmatic + agent (drop stepped, moved to R24/#29)
Issue #12 was rewritten with three structural changes:
1. The iterative-supervision pattern (libero_mujoco_stepped.py)
moves out of R5 entirely. Rationale: the public
nvidia/GR00T-N1.7-LIBERO checkpoint is finetuned end-to-end on
each LIBERO suite, so an in-distribution iterative-supervision
demo is theater — System-2 has nothing to actually decide. The
pattern earns its complexity in OOD scenarios; tracked as R24
(#29).
2. The 'agent' file is back, but reframed: it now demos the
natural-language entry point the deleted libero_example.py
shipped pre-rescope. Agent(tools=[sim, gr00t_inference]); for
--policy=groot the agent itself starts and stops the GR00T
inference service via natural-language prompts to the
gr00t_inference tool — no scripted gr00t_inference(action='start')
from Python.
3. nvidia/GR00T-N1.7-LIBERO is a tree of four sub-checkpoints
(libero_spatial/, libero_10/, libero_object/, libero_goal/);
--task auto-derives which subfolder to download and serve.
Files touched
-------------
DELETED examples/libero_mujoco_stepped.py — moved to R24/#29.
REWRITE examples/libero_mujoco.py:
- New --task <benchmark_name> flag; suite auto-derived from the
task ID's second segment (libero-<suite>-<stem> → libero_<suite>).
- Three-step GR00T setup in the docstring: (1) hf download
--include 'libero_<suite>/*', (2) start service against the
subfolder, (3) run eval. Both the Strands gr00t_inference tool
start and the bare-Docker fallback are shown.
- data_config='libero' in the policy_config (was 'libero_panda');
matches the --data-config libero flag the GR00T service is
started with. Comment notes the libero_panda fallback if the
local Gr00tPolicy registry doesn't recognise the bare 'libero'.
- Filename convention now encodes --task=<benchmark_name>
alongside policy / n_eps / seed; preserves the
rollouts/YYYY_MM_DD/ layout from the deleted SimEnv.
- Spec's default --task 'libero-spatial-pick_up_the_red_cube'
isn't a real LIBERO task. Code keeps it as the documented
default but falls back to the first registered task with a
clear NOTE printed; user-supplied unknown tasks still error
loudly. Documented in the docstring.
- Print line gains a task=<...> field for R15's grep.
NEW examples/libero_mujoco_agent.py:
- Strands Agent(tools=[sim, gr00t_inference]).
- For --policy=groot the agent itself runs the
hf-download → gr00t_inference(action='start') → eval →
gr00t_inference(action='stop') sequence based on natural-language
prompts; no scripted Python invocation of gr00t_inference.
- Single eval prompt explicitly names the libero_<suite>
subfolder so the agent loads the right checkpoint.
- Same --task / --policy / --port / --n-episodes / --seed
interface as the programmatic file.
- Includes a commented-out multi-turn follow-up prompt
showing how the same agent compounds context across calls
(variance vs systematic-failure reasoning).
UPDATE examples/README.md:
- 'Two execution patterns' section now framed as
programmatic vs agent (was one-shot vs iterative).
- Iterative-supervision pattern is explicitly out-of-scope
here with a pointer to R24 / #29 + upstream U6 / #136.
- 'Two policy choices' section explains the GR00T-N1.7-LIBERO
sub-checkpoint tree and how --task auto-picks the subfolder.
- Backend matrix wall-time format kept at 'TBD @ TBD (groot)';
smoke note refers to libero_spatial/.
UPDATE examples/MIGRATION.md:
- Mapping table now has three rows for the three legacy
patterns: SimEnv programmatic → libero_mujoco.py;
agent('Run the task ...') (deleted libero_example.py) →
libero_mujoco_agent.py; SteppedSimEnv → R24/#29 + U6/#136.
- 'Iterative control' section rewritten to point at R24 + U6
instead of an in-repo runnable file (which no longer exists).
- side-by-side After snippet uses data_config='libero' to match
the new spec.
Verified locally
----------------
python examples/libero_mujoco.py --policy mock --n-episodes 5 --seed 42
NOTE: default --task 'libero-spatial-pick_up_the_red_cube' isn't
in real LIBERO ...; falling back to first registered task ...
benchmark_name=libero-spatial-pick_up_the_black_bowl_between_...
policy=mock task=libero-spatial-pick_up_the_black_bowl_between_...
success_rate=0.00 wall_time=0.4s videos=rollouts/...
python examples/libero_mujoco.py --task libero-spatial-pick_up_the_black_bowl_from_table_center_...
benchmark_name=libero-spatial-pick_up_the_black_bowl_from_table_center_...
policy=mock task=... success_rate=0.00 wall_time=0.3s videos=...
python examples/libero_mujoco.py --task libero-spatial-bogus
RuntimeError: --task 'libero-spatial-bogus' is not in the
libero_spatial suite. Available: [...] ← strict on user input
Agent file syntax + imports verified; full end-to-end agent run
needs Bedrock so wasn't executed on this dev box.
(MP4 itself was skipped on this headless dev box because the GLFW
probe fails — MuJoCo rendering unavailable warning. On a machine
with a display the filename pattern resolves as printed.)
Builds on the prior rewrite (31bbbf2) on this branch.
* fix(libero_mujoco): correct --policy=groot setup based on verified n1.7 build
Verified locally on this dev box (NVIDIA L4, 23 GB VRAM, Docker 29.4):
1. Built the upstream n1.7-release container locally
(`docker/build.sh` from NVIDIA/Isaac-GR00T at the n1.7-release
tag — there's no published `nvcr.io/nvidia/isaac-gr00t:latest`
image, the docstring's prior reference was wrong).
2. Ran `python -m gr00t.eval.run_gr00t_server` with
`--model-path /data/checkpoints/GR00T-N1.7-LIBERO/libero_10
--embodiment-tag libero_sim --use-sim-policy-wrapper` —
model loaded in ~80 s (~6 GB GPU mem) and served on port 8000.
The N1.7 server is `gr00t.eval.run_gr00t_server`, NOT the
older `scripts/inference_service.py --server` entrypoint.
3. `Gr00tPolicy(data_config="libero_panda")` connects to the
server. NB: the strands-robots client-side data_config key is
`libero_panda` (registered in `policies/groot`), separate
from the server's `libero_sim` embodiment tag (which aliases
to `EmbodimentTag.LIBERO_PANDA`). Fixed the example to use
`libero_panda` on the client side; the bare `libero` was
rejected by the local registry.
Step-by-step setup in the docstring is now what actually works:
- `docker/build.sh` from upstream n1.7-release → `gr00t:latest`
- `hf download nvidia/GR00T-N1.7-LIBERO --include 'libero_<suite>/*'`
- `docker run -d --gpus all --ipc=host -e HF_TOKEN -v ... gr00t`
(HF token + cache mount required because the VLM backbone
`nvidia/Cosmos-Reason2-2B` is gated)
- `docker exec -d gr00t python -m gr00t.eval.run_gr00t_server ...`
- Run this example with `--policy=groot --port=8000`
Also adds a 'Verification status' section to the docstring listing
the upstream gaps that block end-to-end `--policy=groot` today,
each with a one-line repro for the upstream issues I'm filing
separately:
1. `Simulation` backend doesn't auto-load LIBERO BDDL scenes —
no `agentview`/wrist cameras get created, so `video.image` /
`video.wrist_image` keys never reach the server.
2. `Gr00tPolicy._build_service_observation` adds only a batch
dim; the N1.7 server expects (B, T, H, W, C) for video and
(B, T, D) float32 for state.
3. The Strands `gr00t_inference` tool wraps
`scripts/inference_service.py --server` which doesn't exist
in N1.7 anymore; the bare `docker exec` call in the docstring
is what actually works today.
The model itself works (loads, accepts inference if formatted
correctly). The `--policy=groot` matrix-table number stays TBD
in PR #26 pending the upstream gaps merging (filing follow-ups
separately) or a contributor side-stepping them locally.
`--policy=mock` path remains green — re-verified after this edit:
$ python examples/libero_mujoco.py --policy mock --n-episodes 3
benchmark_name=libero-spatial-pick_up_the_black_bowl_between_…
policy=mock task=… success_rate=0.00 wall_time=0.3s videos=…
* docs(libero_mujoco): cross-reference upstream issue strands-labs/robots#148
The 'Blocked on upstream gaps' appendix in libero_mujoco.py's
Verification status section was previously vague ('Filing as upstream
issue'). All three failures are now tracked together in a single
combined upstream issue: strands-labs/robots#148. Each gap is now
referenced with its position in that issue (Failure 1 / 2 / 3) so a
reader of the example file can land on the right repro section
upstream without fishing.
* docs(libero_mujoco): collapse the manual GR00T server setup; defer to #148
Per the scope decision on PR #26, the GR00T container/checkpoint/
server orchestration is moving upstream into
`strands_robots.tools.gr00t_inference` (tracked as Failure 3 wider on
strands-labs/robots#148). The 50-line docstring block that walked a
user through clone → docker build → hf download → docker run → server
start is now redundant with the issue's 'Reproduction' section.
Replace it with a short pointer:
- one paragraph saying 'this script expects a server on --port; setup
commands live in strands-labs/robots#148, will become a single
gr00t_inference(action="lifecycle", lifecycle="full", ...) call
once that lands';
- a sketch of what that one-line call will look like in this file
after the upstream lands;
- a single-line note pointing at the issue for current manual setup.
Verification status appendix (the 'three upstream gaps still present'
section) is unchanged — that's the part that would lose actionable
content if we deleted it, since it tells a reader running the file
*today* exactly which gaps block --policy=groot end-to-end.
File goes from ~280 lines to 244 lines. --policy=mock smoke re-verified:
policy=mock task=libero-spatial-pick_up_the_black_bowl_…
success_rate=0.00 wall_time=0.3s videos=rollouts/…
* feat(libero_mujoco): wire up gr00t_inference lifecycle for --policy=groot
Now that the upstream catch-up work from strands-labs/robots#148 has
landed (#149 / #150 / #151 / #152 + #155 for the --server flag bug),
'--policy=groot' uses the new gr00t_inference(action='lifecycle',
lifecycle='full', ...) tool to orchestrate the full container +
checkpoint + server setup automatically.
Changes
-------
- Adds --auto-server (default on) which calls the lifecycle tool to:
build the n1.7 container if missing → download
nvidia/GR00T-N1.7-LIBERO/<suite>/ if missing → start the container →
start the inference server. Idempotent on re-runs.
- Adds --no-auto-server escape hatch for users managing their own
service (multi-eval session, custom config, etc.).
- Adds --image / --container / --checkpoint-dir overrides so the
defaults stay sensible without hard-coding paths.
- Tears down via gr00t_inference(action='lifecycle',
lifecycle='teardown', ...) in the finally block so a Ctrl-C or
exception leaves no orphan container.
- Wait-for-model-readiness loop after lifecycle returns: polls GPU
memory until it crosses a 10 GB threshold (model is fully loaded).
The lifecycle tool's 'success' just means the port is bound, the
model loads asynchronously after that — without this loop a fast
evaluate_benchmark race-hangs the first inference request.
- Sets groot_version='n1.7' in policy_config explicitly. Auto-detection
only works when the upstream gr00t package is installed client-side;
when only strands-robots is installed (the common downstream case)
the client defaults to n1.5 wire format and the n1.7 server rejects.
- Sets data_config='libero_panda' on the client (matches the registered
key in policies/groot/DATA_CONFIG_MAP); separate from the server's
--embodiment-tag libero_sim (an alias of LIBERO_PANDA per the
checkpoint's embodiment_id.json).
- Path correction: the lifecycle tool mounts hf_local_dir →
/data/checkpoints, with the HF download placing <suite>/... directly
under that. So the in-container checkpoint path is
/data/checkpoints/<suite>, NOT /data/checkpoints/GR00T-N1.7-LIBERO/<suite>.
Docstring update: replaces the 50-line manual setup block with the
auto-orchestration UX. The 'Verification status' appendix is rewritten
to reflect the current state — three of the four #148 gaps fixed and
verified end-to-end on this dev box; the remaining one
(state-bridging from joint angles → end-effector pose) is filed as
strands-labs/robots#156.
Verified locally (NVIDIA L4):
$ python examples/libero_mujoco.py --policy mock --n-episodes 2
policy=mock task=... success_rate=0.00 wall_time=0.4s videos=...
$ python examples/libero_mujoco.py --policy groot --port 8000 \
--task libero-10-... --n-episodes 2 \
--checkpoint-dir /home/ubuntu/workspace/groot-checkpoints/GR00T-N1.7-LIBERO
[setup] GR00T ZMQ service started on port 8000 (server: n1.7)
[setup] GR00T model loaded (gpu_mem=12712 MiB)
...
RuntimeError: Server error: State key 'state.x' must be in observation
# ↑ Tracked as strands-labs/robots#156. Lifecycle automation, video
# pipeline, n1.7 wire format all work — only state-bridging blocks
# actual eval. Total wall-time: 25 seconds (cached image + checkpoint).
$ python examples/libero_mujoco.py --policy groot --no-auto-server ...
# Same final state; user manages container/server lifetime themselves.
File: 326 → 365 LOC (the +39 is the lifecycle wiring, readiness loop,
and the four new flags).
* feat(libero_mujoco): land --policy=groot end-to-end measurement
All upstream gaps from strands-labs/robots#148 are now resolved:
#147 case-insensitive BDDL parsing (already merged)
#149 N1.7 wire format - (B, T, ...) shape + float32 state (already merged)
#150 N1.7 docker exec target + flags (already merged)
#151 install image/wrist_image cameras for libero_panda (already merged)
#152 full container lifecycle for gr00t_inference (already merged)
#155 drop bogus --server flag from #150's n1.7 builder (just merged)
#161 bridge end-effector FK to state.x/y/z/.../gripper (just merged, closed #156)
#162 pack state.gripper as 2-element array (training shape) (filed)
With #162 applied (one-line gripper shape fix), the example now runs
end-to-end. Measurement on this dev box (NVIDIA L4 / Docker / warm
cache):
benchmark_name=libero-10-LIVING_ROOM_SCENE5_put_the_white_mug_…
policy=groot task=… success_rate=0.00 wall_time=309.6s
videos=rollouts/2026_05_16/…--policy=groot__default.mp4
309.6 s / 5 episodes = ~62 s per episode. success_rate=0.00 is
expected and consistent with the still-open LIBERO scene-loading gap
(load_libero_suite registers BDDL goal predicates but doesn't load
the LIBERO MJCFs because the libero pip package doesn't ship them).
The wall-time IS authoritative for the engine + policy + I/O round-
trip; real success rate waits on a procedural BDDL → MJCF path.
Files updated:
- README.md backend matrix mujoco | 1 cell:
TBD @ TBD (groot)* → ~62 s/ep @ 0.00 (groot, no LIBERO scene)*
Footnote rewritten to cite the actual measurement, the upstream
PRs that unblock it, and the still-open scene-loading caveat.
- examples/README.md mujoco row + smoke note: same pattern. Smoke
paragraph cites the actual measurement instead of saying 'drops in
once a contributor measures'.
- examples/libero_mujoco.py 'Verification status' appendix:
rewritten from a 30-line 'Verified locally' / 'Blocked on upstream
gaps' two-section structure into a 20-line 'Pipeline runs end-to-
end' section with a single 'one open gap' caveat about scene
loading. The earlier list of three upstream gaps is now redundant
(all merged) so it's gone.
This is the first authoritative wall-time number for R15's matrix
flagship. Rest of the matrix (newton / isaac rows) stays TBD as
expected.
* feat(libero_mujoco): land --policy=groot against real LIBERO scenes (post #165)
#165 procedurally generates LIBERO BDDL → MJCF via the upstream libero
package's scene generator inside LiberoAdapter.on_episode_start, so the
policy now operates against the actual trained world (mug / plate /
living-room geometry) instead of a bare Panda + jitter. Closes the
last gap from the eight-PR upstream catch-up wave.
Two example-side adjustments needed to match the new flow:
- Pre-add the robot as 'robot' (LIBERO/RoboSuite naming convention,
matching the name the scene MJCF supplies post-load_scene). Was
'panda' — that name is gone after on_episode_start runs scene-load,
and evaluate_benchmark resolves robot_name BEFORE on_episode_start,
so a stale 'panda' broke the post-load camera/state lookups.
- Drop the explicit robot_name='panda' from the evaluate_benchmark
call. base.evaluate_benchmark auto-picks when there's exactly one
robot, and 'robot' (single, post-scene-load) satisfies that.
Plus a comment block in the example that explains both choices so
the next reader doesn't try to 'fix' them.
Verified end-to-end on the L4 / Docker dev box (warm cache):
--policy=mock --n-episodes 2:
success_rate=0.00 wall_time=6.4s (~3.2 s/ep, vs ~0.4 s/ep
pre-#165 — the 2.8 s delta is per-episode scene-gen + load + step
cost; first-time scene-gen is cached on disk so subsequent calls
hit the cache)
--policy=groot --n-episodes 5
--task libero-10-LIVING_ROOM_SCENE5_put_the_white_mug_…:
success_rate=0.00 wall_time=303.5s (~61 s/ep, ~same as pre-#165)
success_rate=0.00 on the groot run is interesting and worth a future
look — the policy is now operating against the actual trained scene
geometry (verified by inspecting the rendered observations), but
isn't satisfying (On mug_1 plate_1) in 5 episodes. Likely some
combination of:
- init-jitter outside the GR00T training distribution (the adapter's
default ±jitter applies a random perturbation each episode that
may drift further than what the checkpoint trained on)
- camera pose drift (the adapter's wrist-camera pose is a static
top-down approximation, not parented to the gripper as in the
real LIBERO setup the checkpoint trained on)
- 5 episodes is small — variance on a 0.5-ish baseline policy can
easily produce 0/5
That's tuning work, not pipeline work, and lives post-R5 (probably
post-R15 flagship matrix).
Files updated:
- examples/libero_mujoco.py: robot rename + robot_name= drop + clarifying
comments. 'Verification status' appendix rewritten to reflect the
full eight-PR catch-up wave (now nine, with #165) being merged and
the pipeline running against real LIBERO scenes.
- README.md backend matrix mujoco | 1 cell:
~62 s/ep @ 0.00 (groot, no LIBERO scene)*
→
~61 s/ep @ 0.00 (groot, real scene)*
Footnote rewrites the 'no scene' caveat as 'pipeline runs against
real scene; 0.00 is policy-behaviour, not pipeline-broken'.
- examples/README.md mujoco row + smoke note: same pattern, plus the
smoke-note line gets an honest pre-/post-#165 timing comparison
(~0.8 s/ep bare → ~3 s/ep with scene).
* fix(libero_mujoco): record from LIBERO 'image' camera (not deleted 'default')
The previous `cameras=["default"]` recording produced a static MP4 — every frame was byte-identical because the LIBERO scene auto-loader from #165 `load_scene()`s during `evaluate_benchmark`, replacing the world wholesale. The pre-load `default` camera was deleted but the recorder kept its stale handle and emitted the same cached frame for the whole run.
Two changes:
1. **Choose the right camera**: `image` (LIBERO's third-person agentview, renamed from `agentview` by #165) is what the policy was trained against, so it's also the right view for inspecting what the policy is actually doing. Falls back to `default` for `--policy=mock` paths that hit the scene-gen ImportError fallback.
2. **Pre-warm the scene before starting the recorder.** `start_cameras_recording` resolves camera names in the live model at recording-start time. `image` only exists *after* `evaluate_benchmark` calls `on_episode_start`, so calling `start_cameras_recording(cameras=['image'])` upfront errors with `Camera 'image' not found. Available: default`. The fix is to pre-call `spec._generate_scene_from_bddl()` + `sim.load_scene()` ourselves, which materialises the LIBERO scene + cameras early; the per-episode reloads inside `evaluate_benchmark` then reuse the cached `scene_path` so the camera names stay stable across them.
Verified: post-fix MP4 has 162 unique frame hashes (was: 1 unique hash for 90 frames). The robot is visibly active in the rendered video.
A separate concern — the 'image' camera view transitions from a correct LIBERO living-room scene at t=0 to a near-uniform yellow/cream view within ~20s, suggesting the camera view degenerates (gripper-occlusion or pose drift) which probably explains the persistent `success_rate=0.00`. Filing as a separate upstream investigation issue; not gating R5.
* fix(libero_mujoco): prewarm scene + record both cameras for --policy=groot
Two related fixes uncovered while debugging the success-rate=0 mystery
from upstream #166 (the investigation that became upstream #168 round 36-44):
1. **Prewarm before redundant-Panda check.** Pre-#168 round 18 the example
called `sim.add_robot("robot", data_config="panda")` after
`load_scene` to keep the pre-flight check happy. That worked when the
LIBERO scene supplied a Panda named "robot" — `add_robot` was a
silent no-op. But `LiberoAdapter.prewarm` (introduced in upstream's
round-13/16/18 fixes) is the canonical way to register the
scene-supplied Panda on the world.robots side BEFORE add_robot's
redundancy check fires. Without prewarm, the redundant `add_robot`
call would recompile the spec, changing `model.nq` away from the
width `init_states[0]` is sized for, breaking
`_apply_canonical_state` (#168 round 18 finding).
Now: call `spec.prewarm(sim)` if the spec exposes it; only fall
through to `add_robot` defensively for non-LIBERO benchmarks
without a `prewarm` hook.
2. **Record both `image` AND `wrist_image` for groot eval.** The agent
trained on both cameras; recording only `image` (third-person
agentview) made it harder to debug what the wrist view (the camera
driving fine grasp manipulation) was actually seeing. Now records
both so post-hoc analysis has full visibility into what the policy
saw at every frame. `--policy=mock` still records only `default`
(the only camera in the bare-Panda fallback path).
Verified: `--policy=mock` smoke unchanged. `--policy=groot --auto-server`
runs with the new prewarm + dual-camera setup; both MP4s land in
rollouts/.
Bigger upstream-side fixes (rounds 36-44 in strands-labs/robots#168)
landing in a follow-up commit on this branch.
* feat(libero_mujoco): add --engine flag + document upstream PR #168 round 36-44
Round 44 of upstream PR #168 verified `success_rate=1.0` against
nvidia/GR00T-N1.7-LIBERO via the new ``LiberoOffScreenRenderEngine``
SimEngine backend (lands as commit 65d2d18 on
strands-labs/robots#168). This commit makes that path runnable from
the R5 example file with a single ``--engine libero_offscreen_render``
flag.
CHANGES TO `examples/libero_mujoco.py`:
* Add ``--engine={mujoco,libero_offscreen_render}`` flag. Default
``mujoco`` preserves the pre-existing behaviour (auto-generated
scene + custom OSC controller, the legacy default for backwards
compatibility). ``libero_offscreen_render`` routes through the new
upstream-aligned backend that wraps NVIDIA's ``OffScreenRenderEnv``.
Both backends implement the same ``SimEngine`` ABC so the script's
rest-of-flow doesn't care which one is in use.
* Route ``Simulation`` construction through ``create_simulation``
when the new engine is selected; legacy default (``Simulation``
AgentTool wrapping ``MuJoCoSimEngine``) preserved on the default
path.
* Gate the scene-prewarm + ``start_cameras_recording`` block to the
legacy ``--engine=mujoco`` path. The new engine has no
``start_cameras_recording`` (its observations come from upstream's
``OffScreenRenderEnv`` which doesn't expose a per-call recorder)
and no separate ``load_scene`` step (the engine constructs its
own ``OffScreenRenderEnv`` lazily inside
``LiberoAdapter._on_episode_start_offscreen``).
* The grep-stable second print-line gains an ``engine=...`` token
so R15's subprocess-and-parse can disambiguate which backend
produced a row.
* Docstring's Verification-status section rewritten with a 3-row
measurement table (mock+mujoco / groot+mujoco / groot+offscreen+
in-process) plus the round-by-round chronicle of how the 0/5 → 5/5
bisect happened upstream.
CHANGES TO `README.md` (root) + `examples/README.md`:
* Backend matrix gains a ``libero_offscreen_render`` row showing
~14 s/ep @ 1.00 (groot, in-process). Footnote contrasts with the
legacy ``mujoco`` row's ~61 s/ep @ 0.00 (groot, ZMQ client) and
links back to upstream PR #168 round 36-44 for the bisect chronicle.
* Mock-smoke section now lists wall-times for both engines (~3 s/ep
vs ~16 s/ep — the offscreen engine is slower for the mock case
because robosuite constructs the full scene per task vs our cached
procedural generator; the trade-off is byte-equivalent training-
distribution observations on the groot path).
VERIFIED:
* ``python examples/libero_mujoco.py --policy mock --n-episodes 2 --engine mujoco`` → 19.6s, 0.00 (unchanged from pre-edit)
* ``python examples/libero_mujoco.py --policy mock --n-episodes 2 --engine libero_offscreen_render`` → 31.5s, 0.00, no recording (engine doesn't support it; documented in stdout)
* AST parse + smoke import clean
The example file's footprint changed minimally on the legacy path so
existing R15 / matrix-table consumers continue to work without flag
changes; ``--engine=libero_offscreen_render`` is opt-in for groot eval
runs that need ``success_rate>0``.
References:
- Upstream PR #168 (engine + 9 rounds of bisect):
https://github.com/strands-labs/robots/pull/168
- Round 44 breakthrough comment (5/5 success, 73s):
https://github.com/strands-labs/robots/pull/168#issuecomment-4473372219
* docs(libero_mujoco): correct matrix numbers — PR #175 makes mujoco engine 4/5
Yesterday's matrix-cell numbers I added in 6053bd0 were misleading:
* I claimed "--engine=libero_offscreen_render reaches success_rate=1.0
via PR #168" but that path through the example file actually uses our
ZMQ Gr00tPolicy client which still hit the #169 gap → 0/5.
* The 5/5 result I cited came from the diagnostic script
`r44_inprocess_eval.py` which uses NVIDIA's in-process policy
(model loaded directly into Python, no ZMQ). The example file has
no in-process mode.
* That diagnostic script also doesn't call PR #168 round 38's
`_set_eval_seed`, so torch / cuDNN globals drift run-to-run; the
same seed has produced 5/5 and 4/5 in different processes.
Validated 2026-05-19 against PR #175's branch (which closes #169 +
#170 + #171 + #176 — itself layered on top of PR #168 round 36-44):
| --engine | success_rate | wall_time | s/ep |
|---------------------------|--------------|-----------|------|
| mujoco (legacy default) | **0.80** | 168s | ~34 |
| libero_offscreen_render | 0.40 | 389s | ~78 |
| mujoco (pre-PR #175) | 0.00 | 596s | ~119 |
Both are via the ZMQ-client path through the example file (matches
real R15 matrix-table consumer). The mujoco engine OUTPERFORMS the
offscreen engine after PR #175 because PR #175's tuning (state
observation parity, OSC torque parity, gripper home pose, BDDL
`_main` suffix fallback) was specifically targeted at the
`MuJoCoSimEngine` path; the offscreen engine uses upstream's
untuned `OffScreenRenderEnv`.
CHANGES:
* `examples/libero_mujoco.py`: rewrote the Verification-status
appendix with the corrected 3-row table + non-determinism note
+ chronological list of upstream PRs that close the gaps
(PR #168 rounds 36-44 → PR #172 #169 → PR #173 #170 → PR #175
#171 + #176).
* `README.md` (root): matrix-cell `mujoco` row updated to ~34 s/ep
@ 0.80 (was ~3 s/ep mock / ~61 s/ep groot 0.00). Added
`libero_offscreen_render` row at ~78 s/ep @ 0.40. Footnotes link
to PR #168 + PR #175 for the chronicle.
* `examples/README.md`: matrix-cell numbers + footnotes mirror the
root README. Acceptance criterion clarified: `success_rate > 0`,
not a specific number, because of the documented run-to-run
variance.
DEPENDENCIES: this PR depends on PR #168 (squash-merged at upstream
`34f8c37`) + PR #172 + PR #173 + PR #175 (currently open). The
numbers cited won't reproduce on a `strands-robots` install that
doesn't include all four. The example file STILL works on
`--policy=mock` regardless.
Mock numbers unchanged (~3 s/ep mujoco, ~16 s/ep offscreen).
* revert(libero_mujoco): drop --engine flag — upstream #186 retired LiberoOffScreenRenderEngine
After [#186](https://github.com/strands-labs/robots/pull/186) merged on
upstream main (closes #178), the `LiberoOffScreenRenderEngine` package
no longer exists. The `--engine=libero_offscreen_render` option I added
in 6053bd0 (PR #168 round 43 alignment) would now raise a registry
error against any current `strands-robots` install. Per #186: the
`MuJoCoSimEngine` is now byte-equivalent to upstream LIBERO (model-
level inertias, mj_step divergence 0 over 200+ substeps, mean
success_rate=0.92 vs offscreen 0.72, strictly ≥ offscreen on every
(seed, episode)).
CHANGES TO `examples/libero_mujoco.py`:
* Drop the `--engine={mujoco,libero_offscreen_render}` argparse arg.
* Drop the `create_simulation` import + the engine-routing block.
Restore the simpler `Simulation(tool_name="libero_sim", mesh=False)`
construction that PR #26 had pre-6053bd0.
* Drop the engine-conditional gating around scene-prewarm + camera
recording (always runs now, since there's only one engine).
* Drop the `engine=...` token from the second grep-stable print line
(was a one-shot disambiguator added in 6053bd0; redundant now).
* Drop the `--engine=libero_offscreen_render` example from the Usage
section.
* Rewrite the Verification-status appendix:
- Removed the 3-row mock/groot/offscreen+in-process measurement
table (the offscreen row was always misleading per yesterday's
correction; the mujoco row is now the only one and matches
`policy=groot` at ~54 s/ep @ ~0.60-0.92).
- Documents the full upstream catch-up wave: #168 → #172 → #173 →
#175 → #180 → #184 → #186.
- Notes the run-to-run variance bounds (0.60 / 0.80 / 0.92 in
different processes); acceptance is `success_rate > 0`, not a
specific number.
CHANGES TO `README.md` (root) + `examples/README.md`:
* Drop the `libero_offscreen_render` row from the matrix table.
* Update the `mujoco` row's wall-time + success-rate to ~54 s/ep @
0.60-0.92 (groot), measured 2026-05-21 against current upstream main.
* Update the footnote to cite the full upstream catch-up wave (#168 →
#186) and PR #186's reported mean success_rate=0.92.
VERIFIED:
* `--policy=mock --n-episodes 2 --seed 42` → wall=27.4s, success=0.00,
MP4 written under rollouts/2026_05_21/.
* `--policy=groot --no-auto-server --port 8000 --task libero-10-…SCENE5
--n-episodes 5 --seed 42` → wall=267.9s, success=0.60 (3/5), MP4
pair written.
* AST parse + import smoke clean.
DEPENDENCIES: `--policy=groot` numbers reproduce only on a
`strands-robots` install that includes the full upstream catch-up
through #186 (currently merged on upstream main as of 2026-05-21).
The example file still works on `--policy=mock` against any
`strands-robots` install with [sim-mujoco,benchmark-libero] extras.
* feat(libero_mujoco): close ZMQ gap via PR #188 — success_rate=1.00 (5/5)
Validated 2026-05-21 against [`strands-labs/robots#188`](https://github.com/strands-labs/robots/pull/188)
(closes #187, layered on top of all the earlier upstream fixes #168 →
#172 → #173 → #175 → #180 → #184 → #186). PR #188's three coordinated
fixes are:
1. **Spec-driven instruction fallback** — the dominant cause of the
ZMQ-side gap. `examples/libero_mujoco.py` calls
`sim.evaluate_benchmark(...)` without `instruction=`, so the empty
string propagated all the way to the language-conditioned GR00T
policy as `annotation.human.action.task_description=[""]`. PR #188
adds `BenchmarkProtocol.instruction` + an `_evaluate_with_spec`
fallback so spec-supplied instructions reach the policy by default.
2. **Per-episode `policy.reset(seed=)` plumbing** for SERVICE-mode
reproducibility (mirrors PR #180's in-process `set_eval_seed` flow).
3. **Server-side determinism wrapper** (this commit) — drop-in
docker-mountable wrapper that sets `cudnn.deterministic=True` +
`CUBLAS_WORKSPACE_CONFIG=":4096:8"` server-side and applies the
per-episode seed PR #188's client-side plumbing forwards. Optional;
only needed for bit-exact reproducibility.
Both seeds 42 and 100 hit `success_rate=1.00 (5/5)` in ~44 s through
this PR's example file (faster than upstream's reported 52 s for seed
42, and decisively beating the 4/5 / 224 s for seed 100):
| seed | success_rate | wall_time |
|------|--------------|-----------|
| 42 | 1.00 (5/5) | 44.8 s |
| 100 | 1.00 (5/5) | 44.3 s |
Pre-#188 (commit `92d30b3`) the same example produced 0.20-0.60
across runs because the policy was getting empty instructions.
CHANGES:
* `examples/libero_mujoco.py` Verification-status appendix updated
with the post-#188 numbers (5/5 / ~9 s/ep) + chronicle of the
upstream catch-up wave including #188 + a section on the optional
server-side determinism wrapper.
* `README.md` (root) backend-matrix `mujoco` row updated to ~9 s/ep
@ 1.00 (was ~54 s/ep @ 0.60-0.92). Footnote cites #188 as the
unblocker that closed the ZMQ-side gap.
* `examples/README.md` matches root README.
* `examples/gr00t_server_deterministic_wrapper.py` — drop-in
docker-mountable wrapper for users who need bit-exact CUDA
determinism. The example file's docstring references it but works
WITHOUT it (verified at 5/5 above). Set
`STRANDS_GR00T_SERVER_SEED=<int>` to override the default seed.
VERIFIED:
* `examples/libero_mujoco.py --policy groot --port 8000 --task libero-10-…SCENE5
--n-episodes 5 --seed 42` → 5/5 in 44.8s (auto-server, no wrapper)
* Same with `--seed 100` → 5/5 in 44.3s
* AST parse + import smoke clean
DEPENDENCIES: `--policy=groot` numbers reproduce only on a
`strands-robots` install that includes the full upstream catch-up
through #188. The example file still works on `--policy=mock` against
any `strands-robots` install with [sim-mujoco,benchmark-libero] extras.
* refactor(examples): move LIBERO scripts to examples/libero/ + fix data_config bug
Addresses sundargthb's review feedback on PR #26:
1. **Subfolder reorganization** (#26 review). Move LIBERO-specific
scripts under `examples/libero/` so the top-level `examples/` doesn't
get crowded as R8/R12/R23 PRs add their own backend examples:
examples/libero_mujoco.py → examples/libero/run_mujoco.py
examples/libero_mujoco_agent.py → examples/libero/run_mujoco_agent.py
examples/gr00t_server_deterministic_wrapper.py → examples/libero/gr00t_server_deterministic_wrapper.py
IMPORTANT: kept the `run_` prefix instead of the originally-suggested
bare `mujoco.py` / `mujoco_agent.py` names because Python's
sys.path[0]-prepend behaviour means `python examples/libero/mujoco.py`
would shadow the installed PyPI `mujoco` package. The shadow caused
`mujoco.MjSpec()` to fail deep inside `strands_robots` since
`import mujoco` resolved to our example file. `run_mujoco.py` keeps
the subfolder grouping sundargthb wanted while sidestepping the
import collision. Sibling files inside `libero/` cross-reference
each other by relative module name (`run_mujoco.py` ↔
`run_mujoco_agent.py`) so future R8/R12 examples can sit next to
them under `libero/run_isaac.py` etc. without colliding with
`isaac` or other PyPI package names.
2. **`data_config` mismatch fix** (sundargthb's inline comment on
line 123 of the agent file). The agent prompt specified
`data_config='libero'` while the programmatic file uses
`data_config='libero_panda'`. Only `libero_panda` is in
`DATA_CONFIG_MAP`, so the agent flow would KeyError at policy
construction with no useful hint — the LLM passed exactly what we
told it to. Fixed both occurrences in `run_mujoco_agent.py`
(lines 114 and 123) so the agent file matches the programmatic
file.
3. **All cross-references updated** — README.md (root) backend matrix,
examples/README.md table + section pointers, examples/MIGRATION.md
markdown link URLs, in-file docstring usage examples + sibling
refs, and the wrapper script's own mount-path doc.
VERIFIED:
* `python examples/libero/run_mujoco.py --policy mock --n-episodes 2 --seed 42`
→ wall=27.6s, success=0.00, MP4 written.
* AST parse + import smoke clean for all three moved files.
* `mujoco.MjSpec()` no longer collides — confirmed by the smoke
passing where it previously failed.
The post-#188 success_rate=1.00 (5/5) measurement on
`libero-10-LIVING_ROOM_SCENE5_…` (verified yesterday on commit
`a480b15`) carries forward unchanged — the move is path-only, the
example contents are byte-identical aside from docstring path
references.
* docs(examples): trim historical PR-link chains per cagataycali review
Removed the upstream catch-up wave PR chain (#168, #172, #173, #175,
#180, #184, #186, #188) from the verification sections of both
examples/README.md and examples/libero/run_mujoco.py docstring.
Kept the practical reproduction info: measured numbers, hardware,
seeds, suite/task, optional determinism-wrapper section. Forward-
pointing nav (R8/#15, R12/#19, etc.) and operational hints
(`#147`, `#148`, `#165`) preserved — different in nature from
the historical chain cagataycali flagged.
Addresses:
- examples/README.md L90 review comment
- examples/libero/run_mujoco.py L70 review comment
* fix(libero/agent): make --policy=groot work end-to-end
The agent file shipped broken on its advertised --policy=groot mode.
Symptom: 5 distinct failures stacked, eval never produced an MP4 with
real rollout content.
Root cause: too much was delegated to the LLM. The agent thrashed
through 25 tool calls trying to pick the right action from
Simulation's 64-action enum, kept verb-matching 'run' -> run_policy
or 'eval' -> eval_policy (both skip the LIBERO observation adapter,
trigger 'State key state.x must be in observation' server-side), and
along the way redownloaded a cached checkpoint, spawned a duplicate
container ignoring --port, and picked LeRobot Dataset recorder by
mistake (which crashed on the existing rollouts/ dir).
Fix: move deterministic plumbing into the script, leave the LLM-
suited bits to the agent.
Script now owns:
- GR00T container lifecycle (start, wait-for-load, teardown) via the
same gr00t_inference(action='lifecycle', ...) block as run_mujoco.py
- LIBERO scene pre-warm (spec._generate_scene_from_bddl ->
sim.load_scene -> spec.prewarm). Without this, GR00T server rejects
the first observation with 'Video key video.image must be in
observation' because cameras aren't registered yet.
- MP4 recording via start_cameras_recording / stop_cameras_recording
(NOT LeRobot Dataset)
- Default-task fallback for the aspirational placeholder
libero-spatial-pick_up_the_red_cube
- Argparse expanded to mirror run_mujoco.py: --auto-server /
--no-auto-server, --image, --container, --checkpoint-dir
Agent now owns: one tool call (evaluate_benchmark with kwargs from
prompt context) + the natural-language summary. Prompt explicitly
names the action so the LLM can't verb-match its way to a sibling
sub-tool.
Validation on L4 / Docker dev box, libero-10/SCENE5, seed 42:
- --policy=groot, 5 eps: success_rate=1.00 (5/5), wall=154.2s, MP4
1.3 MB image / 2.7 MB wrist (matches run_mujoco.py's 1.7 MB / 3.4 MB
proportionally, eval portion is the same; agent overhead ~100s for
LLM round-trips)
- --policy=mock, 2 eps: success_rate=0.0 (expected), wall=155.4s, MP4
written
Filename now includes --agent suffix
(--policy=groot--agent__image.mp4) so post-hoc analysis can tell
agent-driven from programmatic rollouts at a glance.
Out of scope (filed separately if needed): an upstream
asyncio.run() layering issue in strands-labs/robots
_async_utils._resolve_coroutine — the executor fallback at line 29
papers over it for evaluate_benchmark, but it surfaces noisily in
tracebacks and would benefit from a proper fix in that repo.
* fix(libero/agent): wire synchronous recording (closes upstream #191)
Upstream strands-labs/robots#192 (merged at 974a8f6) added the
synchronous-recording API I filed in #191:
- evaluate_benchmark now accepts on_frame=
- start_cameras_recording_synchronous returns (on_frame, finalize)
closures instead of spawning a daemon recorder thread
Wire the agent file to the new path so its rollout videos no longer
suffer from the daemon-thread / mjData race that produced 4-second
truncated MP4s with greenish GL clear-colour artifacts under
multi-threaded eval (Strands Agent worker thread + daemon recorder).
Shape: @tool wrapper inside main() that captures sim/video_dir/
rec_name and on_frame_cb from the outer scope. Agent picks the
wrapper, fills its kwargs from natural language, returns. Two extra
bits the upstream docstring's example glosses over but matter in
practice:
1. Explicit width=640, height=480 on
start_cameras_recording_synchronous. Without these the eval
thread's renderer cache produces variable-shape arrays in the
buffer that crash imageio's FFmpeg encoder mid-stream with
BrokenPipeError + glibc malloc_consolidate noise.
2. finalize_cb invoked from the script main thread (in a try/finally
around the agent call), NOT from inside the @tool wrapper. The
wrapper runs on the Strands worker thread; FFmpeg subprocess.Popen
spawned from a non-main thread also crashes with BrokenPipeError.
Moving the finalize call to main resolves it.
Validation on L4 / Docker dev box, libero-10/SCENE5:
| Run | wall | success_rate | MP4 |
|---|---|---|---|
| --policy=groot, seed=42, 5 eps | 338 s | 1.00 (5/5) | 5.9 MB image / 13 MB wrist; 1135 frames |
| --policy=groot, seed=42, 1 ep | 75 s | 1.00 (1/1) | 1.2 MB image / 2.4 MB wrist; 227 frames |
| --policy=mock, seed=42, 2 eps | 85 s | 0.00 (expected) | 255 KB default |
The 1135 frames at 5 eps = exactly 1 frame per simulation step
(5 × ~227 steps), matching evaluate_benchmark's per-step on_frame
hook. Compare to the daemon-thread version (cf. PR #26 commit
66005d4) which captured 120 frames in 4 seconds for the same 5-ep
eval — a ~10x improvement in capture rate plus boundary frames are
now clean (no greenish artifact on episode reset).
Sample frames extracted at steps 30 / 100 / 200 / 230 (episode 0->1
boundary) all show the expected scene state with no rendering
artifacts.
Slight wall-time increase vs daemon-thread shape (175 -> 338 s for
5 eps) is the per-step render cost paying for 100% capture instead
of ~2-3% — acceptable trade for matrix-quality video output.
* refactor(libero/agent): drop @tool wrapper, agent picks evaluate_benchmark from Simulation surface
Earlier (b83257d) the agent file wrapped evaluate_benchmark in a
custom @tool to thread upstream PR #192's synchronous recorder
on_frame closure past Strands' tool-dispatch JSON boundary. That
guaranteed bit-exact 1-frame-per-step videos but at the cost of:
- 5-6x wall-time multiplier (per-step render on eval thread)
- ~70 lines of closure-plumbing
- agent picks 1-of-1 wrapper instead of 1-of-64 Simulation actions
— the 'agent picks the right tool' demo aspect was degenerate
Reverting to daemon-thread start_cameras_recording + Agent(tools=[
sim]) lets the agent pick evaluate_benchmark from the full Simulation
action enum via natural language. The prompt explicitly names the
action so the LLM doesn't verb-match 'run' -> run_policy or 'eval' ->
eval_policy (both skip the LIBERO observation adapter).
Trade-off captured in the docstring: the daemon-thread recorder
races with the eval thread on shared mjData under multi-threaded
Agent dispatch, so frame coverage is ~20% of sim steps and an
occasional greenish GL clear-colour artifact may appear. Users who
need guaranteed-clean video should use run_mujoco.py programmatically;
the agent file's job is the natural-language demo.
Validation on L4 / Docker, libero-10/SCENE5, seed 42:
| Run | wall | success_rate | image MP4 / wrist | frames |
|---|---|---|---|---|
| --policy=groot, 5 eps | 61.3s | 1.00 (5/5) | 1.8 MB / 3.5 MB | 224 / 221 |
| --policy=mock, 2 eps | 36.6s | 0.00 (expected) | 219 KB | — |
Compared to b83257d's wrapper version on the same task/seed:
- 5.5x faster wall-time (61s vs 336s)
- 5x fewer frames (224 vs 1135) — daemon thread starved of GIL
- Sample frames at steps 30/100/200/280 verified clean (no greenish
artifacts) on this run; race risk documented but not triggered
Net diff: -69 lines / +57 lines (line count parity with run_mujoco.py
at 421 / 424 lines).
* review: address @cagataycali's CHANGES_REQUESTED on PR #26
P1 (blocking, doc-drift that breaks copy-paste users):
1. MIGRATION.md L41/L96/L168: s/data_config="libero"/"libero_panda"/.
Bare 'libero' raises KeyError at policy construction (DATA_CONFIG_MAP
only registers libero_panda). Same bug sundargthb caught in the
agent file via cf7f8c6 — was missed in the migration doc.
2. MIGRATION.md L86/L163/L184: s/add_robot("panda"/add_robot("robot"/
plus s/robot_name="panda"/omit/. LIBERO scene MJCFs name their
Panda 'robot' (RoboSuite convention), so picking it here keeps the
resolved robot stable across evaluate_benchmark's on_episode_start
scene reload — same rationale captured in run_mujoco.py:297-302.
3. MIGRATION.md L42 narrative: `Agent(tools=[sim, gr00t_inference])`
-> `Agent(tools=[sim])` and updated the table-cell prose to reflect
the lifecycle-vs-agent split that's actually shipped — script owns
container lifecycle / scene pre-warm / MP4 recording, agent only
invokes evaluate_benchmark.
P2 (fold-in before merge):
4. run_mujoco.py L354: removed unused `import random as _random`
(leftover from pre-simplification of scene pre-warm).
5. pyproject.toml: extended hatch lint targets to `examples/`. Surfaced:
- 1x F401 unused import (item 4, fixed)
- 4x E402 in deterministic wrapper (intentional ordering)
- 1x E501 in deterministic wrapper L88 (123 chars)
- black/isort reformats across all three example files
All resolved; E402 sites annotated with `# noqa: E402 # <reason>`.
P3 (followups, addressed the quick wins):
6. __init__.py: switched `__version__ = "0.2.0"` (hardcoded literal)
to `importlib.metadata.version("strands-robots-sim")` with a
PackageNotFoundError fallback. Single source of truth: the VCS tag.
7. __init__.py: dropped `warnings.warn(message, DeprecationWarning)`
before `raise ImportError`. Under `-W error::DeprecationWarning`
the warning would mask the actionable ImportError.
8. gr00t_server_deterministic_wrapper.py: noqa-annotated 4 E402 sites
and wrapped the 123-char print line.
9. README.md: added a footnote to the `~9 s/ep @ 1.00` matrix cell
noting it's a single-sample on L4 with the variance history.
P3.10 (recorder warmup noise) deferred — follow-up to gate retries on
cameras_ready rather than wall-clock budget.
P4 (style):
- run_mujoco_agent.py: deleted the dead 'Optional follow-up' comment-
with-code block at end of file.
Validation:
- black --check / isort --check / flake8 (max-line-length=120): all
clean across strands_robots_sim/ + examples/
- AST parse on all four touched .py files: ok
- import strands_robots_sim; legacy SimEnv import: clean ImportError
- run_mujoco.py --policy mock --n-episodes 2 --seed 42: success_rate=0.00
(expected mock), wall_time=26.9s, MP4 written
Net diff: 7 files, +75/-95 lines.
---------
Co-authored-by: opencode <opencode@local>
What
Default the SO-101 cuRobo / URDF-sim path to the auto-downloaded
strands-robotsSO-101 cache URDF, so--planner curoboworksout-of-the-box on any box that already ran the MuJoCo demo (or has
internet) — no
--curobo-urdfflag needed.Why
examples/so101_curoboships no assets of its own (pure Python +docs). The default MuJoCo demo auto-downloads the SO-101 model into the
external
~/.strands_robots/assets/robotstudio_so101/cache viastrands_robots.assets, and that download already includes a URDF(
so101_new_calib.urdf) and anassets/mesh dir next to the MJCF.But the cuRobo planner (
planner.py:357) and the URDF-sim path(
controller.py:154) only looked at an expliciturdf_path/SO101_URDF, so they made the user hand-stage a URDF even though aperfectly good one was already on disk — and
make_planner(prefer="auto")silently fell back to the scripted planner whenever no URDF flag was set,
even with cuRobo installed.
How
New shared resolvers in
planner.pywith the documented precedence:urdf_path/asset_pathargument,SO101_URDF/SO101_ASSETenv var,(
resolve_model_dir("so101"), which triggers the same auto-downloadthe MuJoCo path uses).
Wired into:
CuroboMotionPlanner.__init__(urdf_path/asset_path)._curobo_usable()—make_planner(prefer="auto")now selects cuRoboout-of-the-box when it's installed and the cache URDF exists.
controller.pysim-arm URDF resolution — the sim loads the exactURDF cuRobo plans with (identical joint conventions + EE frame).
plan_curobo_offline.py+app.pyCLI help text + README.Returns
Nonewhen none of the three resolve, so callers keep theirexisting fail-with-hint behaviour on a box with neither a flag nor the
cache. No change to the default MuJoCo + scripted path.
Verification
On this box the resolver now finds the cached asset with no flags:
pytest examples/so101_curobo/smoke_test.py— 4 passed: the existingMuJoCo pick-place smoke + the fix(so101): app.py / record_dataset fails on re-run — FileExistsError on default dataset dir (no overwrite/unique path) #143 re-run test (default path unchanged),
plus two new tests:
test_resolve_so101_urdf_precedence(explicit → env → cache, mocked).test_resolve_so101_urdf_none_when_unavailable.black --check,isort --check-only,flake8all clean.Notes
examples/so101_curobois self-contained: it ships no assets, relies onthe
strands-robotsauto-download cache for the default run, and nowreuses that same cache for the cuRobo URDF instead of making the user
stage one separately.
USD; this PR only addresses the URDF used by cuRobo + the URDF-sim arm.