Skip to content

feat(so101_curobo): default cuRobo/URDF path to the auto-downloaded SO-101 cache URDF#165

Merged
cagataycali merged 1 commit into
strands-labs:mainfrom
yinsong1986:feat/so101-curobo-default-urdf-from-cache
Jun 30, 2026
Merged

feat(so101_curobo): default cuRobo/URDF path to the auto-downloaded SO-101 cache URDF#165
cagataycali merged 1 commit into
strands-labs:mainfrom
yinsong1986:feat/so101-curobo-default-urdf-from-cache

Conversation

@yinsong1986

Copy link
Copy Markdown
Contributor

What

Default the SO-101 cuRobo / URDF-sim path to the auto-downloaded
strands-robots SO-101 cache URDF
, so --planner curobo works
out-of-the-box on any box that already ran the MuJoCo demo (or has
internet) — no --curobo-urdf flag needed.

Why

examples/so101_curobo ships no assets of its own (pure Python +
docs). The default MuJoCo demo auto-downloads the SO-101 model into the
external ~/.strands_robots/assets/robotstudio_so101/ cache via
strands_robots.assets, and that download already includes a URDF
(so101_new_calib.urdf) and an assets/ mesh dir next to the MJCF.

But the cuRobo planner (planner.py:357) and the URDF-sim path
(controller.py:154) only looked at an explicit urdf_path /
SO101_URDF, so they made the user hand-stage a URDF even though a
perfectly good one was already on disk — and make_planner(prefer="auto")
silently fell back to the scripted planner whenever no URDF flag was set,
even with cuRobo installed.

How

New shared resolvers in planner.py with the documented precedence:

  1. explicit urdf_path / asset_path argument,
  2. SO101_URDF / SO101_ASSET env var,
  3. the auto-resolved cache URDF + mesh dir
    (resolve_model_dir("so101"), which triggers the same auto-download
    the MuJoCo path uses).
resolve_so101_urdf(urdf_path=None)   # -> str | None
resolve_so101_asset(asset_path="")   # -> str  ("" when none resolve)

Wired into:

  • CuroboMotionPlanner.__init__ (urdf_path / asset_path).
  • _curobo_usable()make_planner(prefer="auto") now selects cuRobo
    out-of-the-box when it's installed and the cache URDF exists.
  • controller.py sim-arm URDF resolution — the sim loads the exact
    URDF cuRobo plans with (identical joint conventions + EE frame).
  • plan_curobo_offline.py + app.py CLI help text + README.

Returns None when none of the three resolve, so callers keep their
existing fail-with-hint behaviour on a box with neither a flag nor the
cache. No change to the default MuJoCo + scripted path.

Verification

On this box the resolver now finds the cached asset with no flags:

resolve_so101_urdf(None)  -> ~/.strands_robots/assets/robotstudio_so101/so101_new_calib.urdf
resolve_so101_asset("")   -> ~/.strands_robots/assets/robotstudio_so101/assets

Notes

  • This is the usability papercut flagged while auditing whether
    examples/so101_curobo is self-contained: it ships no assets, relies on
    the strands-robots auto-download cache for the default run, and now
    reuses that same cache for the cuRobo URDF instead of making the user
    stage one separately.
  • Out of scope: the Isaac SO-101 USD path (T2) still needs a faithful
    USD; this PR only addresses the URDF used by cuRobo + the URDF-sim arm.

…ed SO-101 cache URDF

The default MuJoCo demo already auto-downloads the SO-101 model into the
external ~/.strands_robots cache (via strands_robots.assets), and that
download ships a URDF (so101_new_calib.urdf) + an assets/ mesh dir
alongside the MJCF. But the cuRobo planner and the URDF-sim path ignored
it and forced the user to hand-supply --curobo-urdf / SO101_URDF even
though a perfectly good URDF was already on disk -- a usability papercut
(the demo "ships no assets" yet still made you stage a URDF for cuRobo).

Add shared resolvers in planner.py with the documented precedence:
explicit arg -> SO101_URDF / SO101_ASSET env -> the auto-resolved cache
URDF + mesh dir (resolve_model_dir("so101"), which triggers the same
auto-download the MuJoCo path uses). Wire them into:

- CuroboMotionPlanner.__init__ (urdf_path / asset_path),
- _curobo_usable() so `make_planner(prefer="auto")` now selects cuRobo
  out-of-the-box when it's installed + the cache URDF exists (previously it
  fell back to scripted unless a URDF flag was set),
- controller.py's sim-arm URDF resolution (so the sim loads the EXACT URDF
  cuRobo plans with),
- plan_curobo_offline.py + app.py CLI help text.

Returns None when none of the three resolve, so callers keep their existing
fail-with-hint behaviour on a box with neither a flag nor the cache. No
change to the default MuJoCo+scripted path.

Adds two unit tests (precedence + None-when-unavailable) to smoke_test.py;
the existing MuJoCo smoke + strands-labs#143 re-run tests still pass. black / isort /
flake8 clean.

@cagataycali cagataycali left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve. This removes a real usability papercut - the example ships no assets of its own yet forced a --curobo-urdf/SO101_URDF flag even though the MuJoCo path already auto-downloads a perfectly good URDF into the cache.

The resolve_so101_urdf / resolve_so101_asset precedence (explicit kwarg -> env -> cache) is well-factored, the fallback _so101_cache_urdf degrades safely (returns (None, "") on any import/resolve failure so callers keep their fail-with-hint behaviour), and wiring controller.py through the same resolver guarantees the sim arm loads the EXACT URDF cuRobo plans against - that shared-resolver invariant is the part that matters most for correctness.

One behavioural note worth flagging for the record (not blocking): _curobo_usable now routes through resolve_so101_urdf, so on a cache miss make_planner(prefer="auto") can trigger resolve_model_dir("so101")'s auto-download as a side effect of planner selection. That is the intended out-of-the-box behaviour per the PR description and is the same download the MuJoCo path already performs, so it is acceptable for an example - just calling it out so future readers know selection can touch the network on a clean box.

Tests properly mock _so101_cache_urdf (no real network in CI) and pin all three precedence levels plus the None fallback. Lint clean, default MuJoCo+scripted path unchanged. Good to merge.

@cagataycali cagataycali merged commit ee96067 into strands-labs:main Jun 30, 2026
2 checks passed
@github-project-automation github-project-automation Bot moved this from In review to Done in Strands Labs - Robots Jun 30, 2026
Joefear pushed a commit to Joefear/robots-sim that referenced this pull request Jul 2, 2026
* feat: add LIBERO-on-MuJoCo baseline example (R5, #12)

Stage 2 of the umbrella (#8). Adds the first row of the backend
comparison matrix — a minimal LIBERO eval against the upstream MuJoCo
backend shipped by 'strands-robots'. Imports only from 'strands_robots',
not 'strands_robots_sim'; this repo just hosts the file so all four
'libero_*.py' siblings (mujoco / isaac / newton / newton_fleet) live
next to each other for the R15 matrix story.

Two flavours per the design discussion on the issue:

- examples/libero_mujoco.py (direct-API, ~78 LOC, deterministic)
  Runs 'sim.evaluate_benchmark(...)' with policy_provider='mock' against
  the first registered LIBERO spatial task. Prints two grep-stable lines
  ('benchmark_name=...' and 'success_rate=...  wall_time=...s') so R15
  can subprocess-and-parse the output for the comparison table. R15
  ingests this file; R4's README placeholder gets the wall-time number
  filled in from a run of this script.

- examples/libero_mujoco_agent.py (Strands-Agent, natural-language, ~90 LOC)
  Headline pedagogy demo of why 'Simulation' being an 'AgentTool' buys
  you anything: an LLM is given the same task in plain English, picks
  the right action sequence (create_world → add_robot → evaluate_benchmark
  → destroy), and summarises. Output is non-deterministic by design;
  R15 does not ingest this file. Requires a configured LLM provider
  (Bedrock by default). Includes a commented appendix showing the
  deterministic 'agent.tool.libero_sim(action=...)' dispatch path for
  CI / scripted runs without an LLM.

- examples/README.md (~83 LOC)
  Backend matrix table (mujoco row filled with measured wall-time;
  isaac / newton / newton_fleet rows TBD with links to R8 / R12).
  Two-flavours section explaining when to pick which file. Install
  instructions including a temporary git-install line for the
  'sim-mujoco' / 'benchmark-libero' extras (they're on
  'strands-labs/robots' main only until the next PyPI release) and a
  pointer to the upstream BDDL-parser PR (strands-labs/robots#147)
  needed for 'load_libero_suite' to actually register tasks.

Measurement (single-CPU dev machine, mock policy):

    benchmark_name=libero-spatial-pick_up_the_black_bowl_between_...
    success_rate=0.00  wall_time=0.8s

Mock policy can't satisfy real goals so success_rate=0 is expected and
correct. The number to remember is the 0.8 s wall-time, which fills in
the 'mujoco | 1 | …' row in R4's README backend matrix once both PRs
land.

Notes / dependencies:

- Blocked-on-during-implementation: strands-labs/robots#147 (case-
  insensitive BDDL parsing). Without that fix 'load_libero_suite'
  registers 0 tasks and this example raises before timing anything. The
  PR was filed today; once it merges the example runs end-to-end as
  shown above. The wall-time reported here was measured against a local
  build of that fix.
- R4 README placeholder fill-in (mujoco | 1 | <seconds>) is intentionally
  NOT touched in this PR — R4 (#11 / PR #25) is still open. Once R4
  merges, a small follow-up commit on this branch will swap 'TBD' for
  '0.8 s'. Otherwise we'd race the merge.
- R15 (the flagship matrix, #22) will subprocess-and-parse the two
  print() lines from libero_mujoco.py — keep that output format stable.

Closes #12 once the upstream parser fix and the R4 README placeholder
fill-in both land.

* docs: fill in mujoco wall-time in README backend matrix (R5 follow-up)

R4's README (PR #25) shipped with placeholder cells for every row of the
backend comparison matrix. Now that R4 has merged, swap the mujoco row's
'TBD (R5)' for the measured ~0.8 s value from PR #26's libero_mujoco.py
run on a single-CPU dev box.

Also tightens the column header to match what we actually measure
instead of the aspirational '50 eps, libero-spatial-pick_up_the_red_cube'
that was in R4 — that exact task name doesn't exist in real LIBERO and
the example uses 10 eps + the first-registered-task pattern. Header now
reads 'first registered LIBERO spatial task, 10 episodes, mock policy',
matching what R5 / R8 / R12 ship and what R15 will aggregate.

Footnote pins the dependency on the still-open upstream parser fix
strands-labs/robots#147 so readers know the task list is currently empty
without it.

Final size: 152 lines / 7.88 KB — under R4's '≤ 200 lines / ≤ 8 KB'
DoD.

* docs(readme): tick Stage 1 boxes, expand LIBERO matrix to 5 rows, even-handed backend bullets

Three small README updates following Stage 1 completion (#9 #10 #11):

1. Mark Stage 1 status checkboxes as done — R2 / R3 / R4 are merged.

2. Rewrite the "Pick X for Y" bullets so Isaac and Newton are framed
   even-handedly. Both support 4096+ envs; the differentiator is
   rendering vs. solver flexibility, not parallelism. Previous wording
   implied Newton owned fleet-scale RL exclusively, which isn't accurate.

3. Replace the LIBERO matrix table with a 5-row version that includes
   the missing `isaac | 4096` row (tracked as R23 / #27 — IsaacLab-style
   fleet example) and adds Renderer / Why use this row / Example / Issue
   columns so each row answers the "why pick this row" question. Drop
   the redundant per-backend bullet list above the table.

Refs #8 (umbrella). Adds tracking link to #27 (R23, Isaac fleet example).

* feat: align R5 with updated #12 spec — two patterns × two policies + MP4

Re-scope per #12's rewrite:

- Two execution patterns (one-shot vs iterative) instead of two
  drivers (direct-API vs natural-language). The agent-driver demo
  is dropped in favour of an iterative-supervision demo that
  replaces the deleted SteppedSimEnv.
- Two policy choices on every file: --policy {mock,groot}. mock
  for CI / no-GPU smoke; groot for real LIBERO runs against the
  public nvidia/GR00T-N1.7-LIBERO checkpoint.
- MP4 recording around every run, written to rollouts/YYYY_MM_DD/
  with the deleted SimEnv's filename convention preserved (now
  also encoding policy=mock|groot).

Files touched:

- examples/libero_mujoco.py — REWRITE. argparse-driven; handles both
  policies; wraps evaluate_benchmark in start_cameras_recording /
  stop_cameras_recording for an MP4-per-run; documents the GR00T
  service-start path (Strands gr00t_inference tool + bare-Docker
  fallback) in the docstring; uses data_config='libero_panda' (the
  registered key — NOT the bare 'libero' the legacy SimEnv used);
  prints the same two grep-stable lines R15 will subprocess-and-parse,
  now with policy= and videos= fields appended.

- examples/libero_mujoco_stepped.py — NEW. Runnable instance of the
  iterative supervision pattern. Same --policy flag interface as the
  one-shot file. Demonstrates start_policy → time.sleep(1/OBSERVE_HZ)
  → get_state → render → (system-2 hook) loop, then stop_policy +
  stop_cameras_recording. Documents the upstream API gaps it works
  around (state envelope vs flat dict, render returning PNG-in-dict
  vs ndarray, missing reset_benchmark public method).

- examples/libero_mujoco_agent.py — DELETED. The agent-driver flow it
  demoed is replaced by the iterative pattern in the new spec.

- examples/README.md — REWRITE. New 'Two execution patterns' +
  'Two policy choices' sections; mujoco wall-time row split into
  groot-authoritative number (TBD until measured) plus a separate
  mock-smoke-test reference paragraph; usage commands cover all four
  combinations (one-shot/stepped × mock/groot).

- examples/MIGRATION.md — UPDATED. Mapping table cells now link the
  runnable example files (SimEnv → libero_mujoco.py, SteppedSimEnv
  → libero_mujoco_stepped.py); side-by-side and iterative snippets
  fixed to match the actual upstream API (policy_config dict instead
  of policy_port, add_robot before evaluate_benchmark, status-envelope
  semantics on get_state/render); new MP4 section covers the
  preserved rollouts/ filename convention and the per-episode
  segmentation upstream gap.

- README.md — mujoco backend-matrix row format changed to
  'wall-time @ success-rate' to match the updated spec; footnote
  notes the canonical number comes from --policy=groot, mock smoke
  is reference-only.

Verified locally:

  python examples/libero_mujoco.py --policy mock --n-episodes 10 --seed 42
    benchmark_name=libero-spatial-pick_up_the_black_bowl_between_...
    policy=mock  success_rate=0.00  wall_time=0.8s  videos=rollouts/...

  python examples/libero_mujoco_stepped.py --policy mock --max-iters 5
    policy=mock  observe_iters=5  videos=rollouts/...

(Both ran against the upstream BDDL fix from
strands-labs/robots#147; MP4 itself was skipped by the recorder
because this dev box has no X11 display, but on a real machine the
filename pattern resolves as printed.)

Upstream gaps documented in the example docstrings + this PR
description, all small enough to file as follow-ups rather than
gate R5:

1. evaluate_benchmark has no record_video= / video_dir= kwargs —
   we wrap with start_cameras_recording externally, getting one
   MP4 per RUN (not per episode). Per-episode segmentation needs
   upstream plumbing.
2. No public reset_benchmark(benchmark_name, seed) — the iterative
   demo skips per-task scene loading and runs against the default
   Panda scene; the supervision-loop pattern is identical regardless.
3. get_state() returns the standard status envelope, not a flat
   {'reward': ...} dict; loop exit is on --max-iters, not reward.
4. render() returns a status dict with PNG bytes inside, not a
   numpy.ndarray; helper _extract_frame_ndarray unpacks if needed.
5. data_config='libero' alias is not in policies/groot's registry;
   the LIBERO Panda key is 'libero_panda'.

Builds on R5 follow-up be662a8 + the consolidation of #28 (a4a21ef),
both already on this branch.

* feat: re-pivot R5 per #12 v3 — programmatic + agent (drop stepped, moved to R24/#29)

Issue #12 was rewritten with three structural changes:

1. The iterative-supervision pattern (libero_mujoco_stepped.py)
   moves out of R5 entirely. Rationale: the public
   nvidia/GR00T-N1.7-LIBERO checkpoint is finetuned end-to-end on
   each LIBERO suite, so an in-distribution iterative-supervision
   demo is theater — System-2 has nothing to actually decide. The
   pattern earns its complexity in OOD scenarios; tracked as R24
   (#29).
2. The 'agent' file is back, but reframed: it now demos the
   natural-language entry point the deleted libero_example.py
   shipped pre-rescope. Agent(tools=[sim, gr00t_inference]); for
   --policy=groot the agent itself starts and stops the GR00T
   inference service via natural-language prompts to the
   gr00t_inference tool — no scripted gr00t_inference(action='start')
   from Python.
3. nvidia/GR00T-N1.7-LIBERO is a tree of four sub-checkpoints
   (libero_spatial/, libero_10/, libero_object/, libero_goal/);
   --task auto-derives which subfolder to download and serve.

Files touched
-------------

DELETED examples/libero_mujoco_stepped.py — moved to R24/#29.

REWRITE examples/libero_mujoco.py:
- New --task <benchmark_name> flag; suite auto-derived from the
  task ID's second segment (libero-<suite>-<stem> → libero_<suite>).
- Three-step GR00T setup in the docstring: (1) hf download
  --include 'libero_<suite>/*', (2) start service against the
  subfolder, (3) run eval. Both the Strands gr00t_inference tool
  start and the bare-Docker fallback are shown.
- data_config='libero' in the policy_config (was 'libero_panda');
  matches the --data-config libero flag the GR00T service is
  started with. Comment notes the libero_panda fallback if the
  local Gr00tPolicy registry doesn't recognise the bare 'libero'.
- Filename convention now encodes --task=<benchmark_name>
  alongside policy / n_eps / seed; preserves the
  rollouts/YYYY_MM_DD/ layout from the deleted SimEnv.
- Spec's default --task 'libero-spatial-pick_up_the_red_cube'
  isn't a real LIBERO task. Code keeps it as the documented
  default but falls back to the first registered task with a
  clear NOTE printed; user-supplied unknown tasks still error
  loudly. Documented in the docstring.
- Print line gains a task=<...> field for R15's grep.

NEW examples/libero_mujoco_agent.py:
- Strands Agent(tools=[sim, gr00t_inference]).
- For --policy=groot the agent itself runs the
  hf-download → gr00t_inference(action='start') → eval →
  gr00t_inference(action='stop') sequence based on natural-language
  prompts; no scripted Python invocation of gr00t_inference.
- Single eval prompt explicitly names the libero_<suite>
  subfolder so the agent loads the right checkpoint.
- Same --task / --policy / --port / --n-episodes / --seed
  interface as the programmatic file.
- Includes a commented-out multi-turn follow-up prompt
  showing how the same agent compounds context across calls
  (variance vs systematic-failure reasoning).

UPDATE examples/README.md:
- 'Two execution patterns' section now framed as
  programmatic vs agent (was one-shot vs iterative).
- Iterative-supervision pattern is explicitly out-of-scope
  here with a pointer to R24 / #29 + upstream U6 / #136.
- 'Two policy choices' section explains the GR00T-N1.7-LIBERO
  sub-checkpoint tree and how --task auto-picks the subfolder.
- Backend matrix wall-time format kept at 'TBD @ TBD (groot)';
  smoke note refers to libero_spatial/.

UPDATE examples/MIGRATION.md:
- Mapping table now has three rows for the three legacy
  patterns: SimEnv programmatic → libero_mujoco.py;
  agent('Run the task ...') (deleted libero_example.py) →
  libero_mujoco_agent.py; SteppedSimEnv → R24/#29 + U6/#136.
- 'Iterative control' section rewritten to point at R24 + U6
  instead of an in-repo runnable file (which no longer exists).
- side-by-side After snippet uses data_config='libero' to match
  the new spec.

Verified locally
----------------

  python examples/libero_mujoco.py --policy mock --n-episodes 5 --seed 42
    NOTE: default --task 'libero-spatial-pick_up_the_red_cube' isn't
    in real LIBERO ...; falling back to first registered task ...
    benchmark_name=libero-spatial-pick_up_the_black_bowl_between_...
    policy=mock  task=libero-spatial-pick_up_the_black_bowl_between_...
    success_rate=0.00  wall_time=0.4s  videos=rollouts/...

  python examples/libero_mujoco.py --task libero-spatial-pick_up_the_black_bowl_from_table_center_...
    benchmark_name=libero-spatial-pick_up_the_black_bowl_from_table_center_...
    policy=mock  task=...  success_rate=0.00  wall_time=0.3s  videos=...

  python examples/libero_mujoco.py --task libero-spatial-bogus
    RuntimeError: --task 'libero-spatial-bogus' is not in the
    libero_spatial suite. Available: [...]   ← strict on user input

  Agent file syntax + imports verified; full end-to-end agent run
  needs Bedrock so wasn't executed on this dev box.

(MP4 itself was skipped on this headless dev box because the GLFW
probe fails — MuJoCo rendering unavailable warning. On a machine
with a display the filename pattern resolves as printed.)

Builds on the prior rewrite (31bbbf2) on this branch.

* fix(libero_mujoco): correct --policy=groot setup based on verified n1.7 build

Verified locally on this dev box (NVIDIA L4, 23 GB VRAM, Docker 29.4):

1. Built the upstream n1.7-release container locally
   (`docker/build.sh` from NVIDIA/Isaac-GR00T at the n1.7-release
   tag — there's no published `nvcr.io/nvidia/isaac-gr00t:latest`
   image, the docstring's prior reference was wrong).
2. Ran `python -m gr00t.eval.run_gr00t_server` with
   `--model-path /data/checkpoints/GR00T-N1.7-LIBERO/libero_10
   --embodiment-tag libero_sim --use-sim-policy-wrapper` —
   model loaded in ~80 s (~6 GB GPU mem) and served on port 8000.
   The N1.7 server is `gr00t.eval.run_gr00t_server`, NOT the
   older `scripts/inference_service.py --server` entrypoint.
3. `Gr00tPolicy(data_config="libero_panda")` connects to the
   server. NB: the strands-robots client-side data_config key is
   `libero_panda` (registered in `policies/groot`), separate
   from the server's `libero_sim` embodiment tag (which aliases
   to `EmbodimentTag.LIBERO_PANDA`). Fixed the example to use
   `libero_panda` on the client side; the bare `libero` was
   rejected by the local registry.

Step-by-step setup in the docstring is now what actually works:

- `docker/build.sh` from upstream n1.7-release → `gr00t:latest`
- `hf download nvidia/GR00T-N1.7-LIBERO --include 'libero_<suite>/*'`
- `docker run -d --gpus all --ipc=host -e HF_TOKEN -v ... gr00t`
  (HF token + cache mount required because the VLM backbone
  `nvidia/Cosmos-Reason2-2B` is gated)
- `docker exec -d gr00t python -m gr00t.eval.run_gr00t_server ...`
- Run this example with `--policy=groot --port=8000`

Also adds a 'Verification status' section to the docstring listing
the upstream gaps that block end-to-end `--policy=groot` today,
each with a one-line repro for the upstream issues I'm filing
separately:

1. `Simulation` backend doesn't auto-load LIBERO BDDL scenes —
   no `agentview`/wrist cameras get created, so `video.image` /
   `video.wrist_image` keys never reach the server.
2. `Gr00tPolicy._build_service_observation` adds only a batch
   dim; the N1.7 server expects (B, T, H, W, C) for video and
   (B, T, D) float32 for state.
3. The Strands `gr00t_inference` tool wraps
   `scripts/inference_service.py --server` which doesn't exist
   in N1.7 anymore; the bare `docker exec` call in the docstring
   is what actually works today.

The model itself works (loads, accepts inference if formatted
correctly). The `--policy=groot` matrix-table number stays TBD
in PR #26 pending the upstream gaps merging (filing follow-ups
separately) or a contributor side-stepping them locally.

`--policy=mock` path remains green — re-verified after this edit:

  $ python examples/libero_mujoco.py --policy mock --n-episodes 3
  benchmark_name=libero-spatial-pick_up_the_black_bowl_between_…
  policy=mock  task=…  success_rate=0.00  wall_time=0.3s  videos=…

* docs(libero_mujoco): cross-reference upstream issue strands-labs/robots#148

The 'Blocked on upstream gaps' appendix in libero_mujoco.py's
Verification status section was previously vague ('Filing as upstream
issue'). All three failures are now tracked together in a single
combined upstream issue: strands-labs/robots#148. Each gap is now
referenced with its position in that issue (Failure 1 / 2 / 3) so a
reader of the example file can land on the right repro section
upstream without fishing.

* docs(libero_mujoco): collapse the manual GR00T server setup; defer to #148

Per the scope decision on PR #26, the GR00T container/checkpoint/
server orchestration is moving upstream into
`strands_robots.tools.gr00t_inference` (tracked as Failure 3 wider on
strands-labs/robots#148). The 50-line docstring block that walked a
user through clone → docker build → hf download → docker run → server
start is now redundant with the issue's 'Reproduction' section.

Replace it with a short pointer:

  - one paragraph saying 'this script expects a server on --port; setup
    commands live in strands-labs/robots#148, will become a single
    gr00t_inference(action="lifecycle", lifecycle="full", ...) call
    once that lands';
  - a sketch of what that one-line call will look like in this file
    after the upstream lands;
  - a single-line note pointing at the issue for current manual setup.

Verification status appendix (the 'three upstream gaps still present'
section) is unchanged — that's the part that would lose actionable
content if we deleted it, since it tells a reader running the file
*today* exactly which gaps block --policy=groot end-to-end.

File goes from ~280 lines to 244 lines. --policy=mock smoke re-verified:

  policy=mock  task=libero-spatial-pick_up_the_black_bowl_…
  success_rate=0.00  wall_time=0.3s  videos=rollouts/…

* feat(libero_mujoco): wire up gr00t_inference lifecycle for --policy=groot

Now that the upstream catch-up work from strands-labs/robots#148 has
landed (#149 / #150 / #151 / #152 + #155 for the --server flag bug),
'--policy=groot' uses the new gr00t_inference(action='lifecycle',
lifecycle='full', ...) tool to orchestrate the full container +
checkpoint + server setup automatically.

Changes
-------

- Adds --auto-server (default on) which calls the lifecycle tool to:
  build the n1.7 container if missing → download
  nvidia/GR00T-N1.7-LIBERO/<suite>/ if missing → start the container →
  start the inference server. Idempotent on re-runs.

- Adds --no-auto-server escape hatch for users managing their own
  service (multi-eval session, custom config, etc.).

- Adds --image / --container / --checkpoint-dir overrides so the
  defaults stay sensible without hard-coding paths.

- Tears down via gr00t_inference(action='lifecycle',
  lifecycle='teardown', ...) in the finally block so a Ctrl-C or
  exception leaves no orphan container.

- Wait-for-model-readiness loop after lifecycle returns: polls GPU
  memory until it crosses a 10 GB threshold (model is fully loaded).
  The lifecycle tool's 'success' just means the port is bound, the
  model loads asynchronously after that — without this loop a fast
  evaluate_benchmark race-hangs the first inference request.

- Sets groot_version='n1.7' in policy_config explicitly. Auto-detection
  only works when the upstream gr00t package is installed client-side;
  when only strands-robots is installed (the common downstream case)
  the client defaults to n1.5 wire format and the n1.7 server rejects.

- Sets data_config='libero_panda' on the client (matches the registered
  key in policies/groot/DATA_CONFIG_MAP); separate from the server's
  --embodiment-tag libero_sim (an alias of LIBERO_PANDA per the
  checkpoint's embodiment_id.json).

- Path correction: the lifecycle tool mounts hf_local_dir →
  /data/checkpoints, with the HF download placing <suite>/... directly
  under that. So the in-container checkpoint path is
  /data/checkpoints/<suite>, NOT /data/checkpoints/GR00T-N1.7-LIBERO/<suite>.

Docstring update: replaces the 50-line manual setup block with the
auto-orchestration UX. The 'Verification status' appendix is rewritten
to reflect the current state — three of the four #148 gaps fixed and
verified end-to-end on this dev box; the remaining one
(state-bridging from joint angles → end-effector pose) is filed as
strands-labs/robots#156.

Verified locally (NVIDIA L4):

  $ python examples/libero_mujoco.py --policy mock --n-episodes 2
  policy=mock  task=...  success_rate=0.00  wall_time=0.4s  videos=...

  $ python examples/libero_mujoco.py --policy groot --port 8000 \
      --task libero-10-... --n-episodes 2 \
      --checkpoint-dir /home/ubuntu/workspace/groot-checkpoints/GR00T-N1.7-LIBERO

  [setup] GR00T ZMQ service started on port 8000 (server: n1.7)
  [setup] GR00T model loaded (gpu_mem=12712 MiB)
  ...
  RuntimeError: Server error: State key 'state.x' must be in observation
  # ↑ Tracked as strands-labs/robots#156. Lifecycle automation, video
  # pipeline, n1.7 wire format all work — only state-bridging blocks
  # actual eval. Total wall-time: 25 seconds (cached image + checkpoint).

  $ python examples/libero_mujoco.py --policy groot --no-auto-server ...
  # Same final state; user manages container/server lifetime themselves.

File: 326 → 365 LOC (the +39 is the lifecycle wiring, readiness loop,
and the four new flags).

* feat(libero_mujoco): land --policy=groot end-to-end measurement

All upstream gaps from strands-labs/robots#148 are now resolved:

  #147   case-insensitive BDDL parsing                         (already merged)
  #149   N1.7 wire format - (B, T, ...) shape + float32 state  (already merged)
  #150   N1.7 docker exec target + flags                       (already merged)
  #151   install image/wrist_image cameras for libero_panda    (already merged)
  #152   full container lifecycle for gr00t_inference          (already merged)
  #155   drop bogus --server flag from #150's n1.7 builder     (just merged)
  #161   bridge end-effector FK to state.x/y/z/.../gripper     (just merged, closed #156)
  #162   pack state.gripper as 2-element array (training shape) (filed)

With #162 applied (one-line gripper shape fix), the example now runs
end-to-end. Measurement on this dev box (NVIDIA L4 / Docker / warm
cache):

  benchmark_name=libero-10-LIVING_ROOM_SCENE5_put_the_white_mug_…
  policy=groot  task=…  success_rate=0.00  wall_time=309.6s
  videos=rollouts/2026_05_16/…--policy=groot__default.mp4

309.6 s / 5 episodes = ~62 s per episode. success_rate=0.00 is
expected and consistent with the still-open LIBERO scene-loading gap
(load_libero_suite registers BDDL goal predicates but doesn't load
the LIBERO MJCFs because the libero pip package doesn't ship them).
The wall-time IS authoritative for the engine + policy + I/O round-
trip; real success rate waits on a procedural BDDL → MJCF path.

Files updated:

- README.md backend matrix mujoco | 1 cell:
  TBD @ TBD (groot)*  →  ~62 s/ep @ 0.00 (groot, no LIBERO scene)*
  Footnote rewritten to cite the actual measurement, the upstream
  PRs that unblock it, and the still-open scene-loading caveat.

- examples/README.md mujoco row + smoke note: same pattern. Smoke
  paragraph cites the actual measurement instead of saying 'drops in
  once a contributor measures'.

- examples/libero_mujoco.py 'Verification status' appendix:
  rewritten from a 30-line 'Verified locally' / 'Blocked on upstream
  gaps' two-section structure into a 20-line 'Pipeline runs end-to-
  end' section with a single 'one open gap' caveat about scene
  loading. The earlier list of three upstream gaps is now redundant
  (all merged) so it's gone.

This is the first authoritative wall-time number for R15's matrix
flagship. Rest of the matrix (newton / isaac rows) stays TBD as
expected.

* feat(libero_mujoco): land --policy=groot against real LIBERO scenes (post #165)

#165 procedurally generates LIBERO BDDL → MJCF via the upstream libero
package's scene generator inside LiberoAdapter.on_episode_start, so the
policy now operates against the actual trained world (mug / plate /
living-room geometry) instead of a bare Panda + jitter. Closes the
last gap from the eight-PR upstream catch-up wave.

Two example-side adjustments needed to match the new flow:

- Pre-add the robot as 'robot' (LIBERO/RoboSuite naming convention,
  matching the name the scene MJCF supplies post-load_scene). Was
  'panda' — that name is gone after on_episode_start runs scene-load,
  and evaluate_benchmark resolves robot_name BEFORE on_episode_start,
  so a stale 'panda' broke the post-load camera/state lookups.

- Drop the explicit robot_name='panda' from the evaluate_benchmark
  call. base.evaluate_benchmark auto-picks when there's exactly one
  robot, and 'robot' (single, post-scene-load) satisfies that.

Plus a comment block in the example that explains both choices so
the next reader doesn't try to 'fix' them.

Verified end-to-end on the L4 / Docker dev box (warm cache):

  --policy=mock --n-episodes 2:
    success_rate=0.00  wall_time=6.4s   (~3.2 s/ep, vs ~0.4 s/ep
    pre-#165 — the 2.8 s delta is per-episode scene-gen + load + step
    cost; first-time scene-gen is cached on disk so subsequent calls
    hit the cache)

  --policy=groot --n-episodes 5
        --task libero-10-LIVING_ROOM_SCENE5_put_the_white_mug_…:
    success_rate=0.00  wall_time=303.5s (~61 s/ep, ~same as pre-#165)

success_rate=0.00 on the groot run is interesting and worth a future
look — the policy is now operating against the actual trained scene
geometry (verified by inspecting the rendered observations), but
isn't satisfying (On mug_1 plate_1) in 5 episodes. Likely some
combination of:

  - init-jitter outside the GR00T training distribution (the adapter's
    default ±jitter applies a random perturbation each episode that
    may drift further than what the checkpoint trained on)
  - camera pose drift (the adapter's wrist-camera pose is a static
    top-down approximation, not parented to the gripper as in the
    real LIBERO setup the checkpoint trained on)
  - 5 episodes is small — variance on a 0.5-ish baseline policy can
    easily produce 0/5

That's tuning work, not pipeline work, and lives post-R5 (probably
post-R15 flagship matrix).

Files updated:

- examples/libero_mujoco.py: robot rename + robot_name= drop + clarifying
  comments. 'Verification status' appendix rewritten to reflect the
  full eight-PR catch-up wave (now nine, with #165) being merged and
  the pipeline running against real LIBERO scenes.

- README.md backend matrix mujoco | 1 cell:
  ~62 s/ep @ 0.00 (groot, no LIBERO scene)*
    →
  ~61 s/ep @ 0.00 (groot, real scene)*
  Footnote rewrites the 'no scene' caveat as 'pipeline runs against
  real scene; 0.00 is policy-behaviour, not pipeline-broken'.

- examples/README.md mujoco row + smoke note: same pattern, plus the
  smoke-note line gets an honest pre-/post-#165 timing comparison
  (~0.8 s/ep bare → ~3 s/ep with scene).

* fix(libero_mujoco): record from LIBERO 'image' camera (not deleted 'default')

The previous `cameras=["default"]` recording produced a static MP4 — every frame was byte-identical because the LIBERO scene auto-loader from #165 `load_scene()`s during `evaluate_benchmark`, replacing the world wholesale. The pre-load `default` camera was deleted but the recorder kept its stale handle and emitted the same cached frame for the whole run.

Two changes:

1. **Choose the right camera**: `image` (LIBERO's third-person agentview, renamed from `agentview` by #165) is what the policy was trained against, so it's also the right view for inspecting what the policy is actually doing. Falls back to `default` for `--policy=mock` paths that hit the scene-gen ImportError fallback.

2. **Pre-warm the scene before starting the recorder.** `start_cameras_recording` resolves camera names in the live model at recording-start time. `image` only exists *after* `evaluate_benchmark` calls `on_episode_start`, so calling `start_cameras_recording(cameras=['image'])` upfront errors with `Camera 'image' not found. Available: default`. The fix is to pre-call `spec._generate_scene_from_bddl()` + `sim.load_scene()` ourselves, which materialises the LIBERO scene + cameras early; the per-episode reloads inside `evaluate_benchmark` then reuse the cached `scene_path` so the camera names stay stable across them.

Verified: post-fix MP4 has 162 unique frame hashes (was: 1 unique hash for 90 frames). The robot is visibly active in the rendered video.

A separate concern — the 'image' camera view transitions from a correct LIBERO living-room scene at t=0 to a near-uniform yellow/cream view within ~20s, suggesting the camera view degenerates (gripper-occlusion or pose drift) which probably explains the persistent `success_rate=0.00`. Filing as a separate upstream investigation issue; not gating R5.

* fix(libero_mujoco): prewarm scene + record both cameras for --policy=groot

Two related fixes uncovered while debugging the success-rate=0 mystery
from upstream #166 (the investigation that became upstream #168 round 36-44):

1. **Prewarm before redundant-Panda check.** Pre-#168 round 18 the example
   called `sim.add_robot("robot", data_config="panda")` after
   `load_scene` to keep the pre-flight check happy. That worked when the
   LIBERO scene supplied a Panda named "robot" — `add_robot` was a
   silent no-op. But `LiberoAdapter.prewarm` (introduced in upstream's
   round-13/16/18 fixes) is the canonical way to register the
   scene-supplied Panda on the world.robots side BEFORE add_robot's
   redundancy check fires. Without prewarm, the redundant `add_robot`
   call would recompile the spec, changing `model.nq` away from the
   width `init_states[0]` is sized for, breaking
   `_apply_canonical_state` (#168 round 18 finding).

   Now: call `spec.prewarm(sim)` if the spec exposes it; only fall
   through to `add_robot` defensively for non-LIBERO benchmarks
   without a `prewarm` hook.

2. **Record both `image` AND `wrist_image` for groot eval.** The agent
   trained on both cameras; recording only `image` (third-person
   agentview) made it harder to debug what the wrist view (the camera
   driving fine grasp manipulation) was actually seeing. Now records
   both so post-hoc analysis has full visibility into what the policy
   saw at every frame. `--policy=mock` still records only `default`
   (the only camera in the bare-Panda fallback path).

Verified: `--policy=mock` smoke unchanged. `--policy=groot --auto-server`
runs with the new prewarm + dual-camera setup; both MP4s land in
rollouts/.

Bigger upstream-side fixes (rounds 36-44 in strands-labs/robots#168)
landing in a follow-up commit on this branch.

* feat(libero_mujoco): add --engine flag + document upstream PR #168 round 36-44

Round 44 of upstream PR #168 verified `success_rate=1.0` against
nvidia/GR00T-N1.7-LIBERO via the new ``LiberoOffScreenRenderEngine``
SimEngine backend (lands as commit 65d2d18 on
strands-labs/robots#168). This commit makes that path runnable from
the R5 example file with a single ``--engine libero_offscreen_render``
flag.

CHANGES TO `examples/libero_mujoco.py`:

* Add ``--engine={mujoco,libero_offscreen_render}`` flag. Default
  ``mujoco`` preserves the pre-existing behaviour (auto-generated
  scene + custom OSC controller, the legacy default for backwards
  compatibility). ``libero_offscreen_render`` routes through the new
  upstream-aligned backend that wraps NVIDIA's ``OffScreenRenderEnv``.
  Both backends implement the same ``SimEngine`` ABC so the script's
  rest-of-flow doesn't care which one is in use.

* Route ``Simulation`` construction through ``create_simulation``
  when the new engine is selected; legacy default (``Simulation``
  AgentTool wrapping ``MuJoCoSimEngine``) preserved on the default
  path.

* Gate the scene-prewarm + ``start_cameras_recording`` block to the
  legacy ``--engine=mujoco`` path. The new engine has no
  ``start_cameras_recording`` (its observations come from upstream's
  ``OffScreenRenderEnv`` which doesn't expose a per-call recorder)
  and no separate ``load_scene`` step (the engine constructs its
  own ``OffScreenRenderEnv`` lazily inside
  ``LiberoAdapter._on_episode_start_offscreen``).

* The grep-stable second print-line gains an ``engine=...`` token
  so R15's subprocess-and-parse can disambiguate which backend
  produced a row.

* Docstring's Verification-status section rewritten with a 3-row
  measurement table (mock+mujoco / groot+mujoco / groot+offscreen+
  in-process) plus the round-by-round chronicle of how the 0/5 → 5/5
  bisect happened upstream.

CHANGES TO `README.md` (root) + `examples/README.md`:

* Backend matrix gains a ``libero_offscreen_render`` row showing
  ~14 s/ep @ 1.00 (groot, in-process). Footnote contrasts with the
  legacy ``mujoco`` row's ~61 s/ep @ 0.00 (groot, ZMQ client) and
  links back to upstream PR #168 round 36-44 for the bisect chronicle.

* Mock-smoke section now lists wall-times for both engines (~3 s/ep
  vs ~16 s/ep — the offscreen engine is slower for the mock case
  because robosuite constructs the full scene per task vs our cached
  procedural generator; the trade-off is byte-equivalent training-
  distribution observations on the groot path).

VERIFIED:
* ``python examples/libero_mujoco.py --policy mock --n-episodes 2 --engine mujoco`` → 19.6s, 0.00 (unchanged from pre-edit)
* ``python examples/libero_mujoco.py --policy mock --n-episodes 2 --engine libero_offscreen_render`` → 31.5s, 0.00, no recording (engine doesn't support it; documented in stdout)
* AST parse + smoke import clean

The example file's footprint changed minimally on the legacy path so
existing R15 / matrix-table consumers continue to work without flag
changes; ``--engine=libero_offscreen_render`` is opt-in for groot eval
runs that need ``success_rate>0``.

References:
- Upstream PR #168 (engine + 9 rounds of bisect):
  https://github.com/strands-labs/robots/pull/168
- Round 44 breakthrough comment (5/5 success, 73s):
  https://github.com/strands-labs/robots/pull/168#issuecomment-4473372219

* docs(libero_mujoco): correct matrix numbers — PR #175 makes mujoco engine 4/5

Yesterday's matrix-cell numbers I added in 6053bd0 were misleading:

* I claimed "--engine=libero_offscreen_render reaches success_rate=1.0
  via PR #168" but that path through the example file actually uses our
  ZMQ Gr00tPolicy client which still hit the #169 gap → 0/5.
* The 5/5 result I cited came from the diagnostic script
  `r44_inprocess_eval.py` which uses NVIDIA's in-process policy
  (model loaded directly into Python, no ZMQ). The example file has
  no in-process mode.
* That diagnostic script also doesn't call PR #168 round 38's
  `_set_eval_seed`, so torch / cuDNN globals drift run-to-run; the
  same seed has produced 5/5 and 4/5 in different processes.

Validated 2026-05-19 against PR #175's branch (which closes #169 +
#170 + #171 + #176 — itself layered on top of PR #168 round 36-44):

| --engine                  | success_rate | wall_time | s/ep |
|---------------------------|--------------|-----------|------|
| mujoco (legacy default)   | **0.80**     | 168s      | ~34  |
| libero_offscreen_render   | 0.40         | 389s      | ~78  |
| mujoco (pre-PR #175)      | 0.00         | 596s      | ~119 |

Both are via the ZMQ-client path through the example file (matches
real R15 matrix-table consumer). The mujoco engine OUTPERFORMS the
offscreen engine after PR #175 because PR #175's tuning (state
observation parity, OSC torque parity, gripper home pose, BDDL
`_main` suffix fallback) was specifically targeted at the
`MuJoCoSimEngine` path; the offscreen engine uses upstream's
untuned `OffScreenRenderEnv`.

CHANGES:

* `examples/libero_mujoco.py`: rewrote the Verification-status
  appendix with the corrected 3-row table + non-determinism note
  + chronological list of upstream PRs that close the gaps
  (PR #168 rounds 36-44 → PR #172 #169 → PR #173 #170 → PR #175
  #171 + #176).

* `README.md` (root): matrix-cell `mujoco` row updated to ~34 s/ep
  @ 0.80 (was ~3 s/ep mock / ~61 s/ep groot 0.00). Added
  `libero_offscreen_render` row at ~78 s/ep @ 0.40. Footnotes link
  to PR #168 + PR #175 for the chronicle.

* `examples/README.md`: matrix-cell numbers + footnotes mirror the
  root README. Acceptance criterion clarified: `success_rate > 0`,
  not a specific number, because of the documented run-to-run
  variance.

DEPENDENCIES: this PR depends on PR #168 (squash-merged at upstream
`34f8c37`) + PR #172 + PR #173 + PR #175 (currently open). The
numbers cited won't reproduce on a `strands-robots` install that
doesn't include all four. The example file STILL works on
`--policy=mock` regardless.

Mock numbers unchanged (~3 s/ep mujoco, ~16 s/ep offscreen).

* revert(libero_mujoco): drop --engine flag — upstream #186 retired LiberoOffScreenRenderEngine

After [#186](https://github.com/strands-labs/robots/pull/186) merged on
upstream main (closes #178), the `LiberoOffScreenRenderEngine` package
no longer exists. The `--engine=libero_offscreen_render` option I added
in 6053bd0 (PR #168 round 43 alignment) would now raise a registry
error against any current `strands-robots` install. Per #186: the
`MuJoCoSimEngine` is now byte-equivalent to upstream LIBERO (model-
level inertias, mj_step divergence 0 over 200+ substeps, mean
success_rate=0.92 vs offscreen 0.72, strictly ≥ offscreen on every
(seed, episode)).

CHANGES TO `examples/libero_mujoco.py`:

* Drop the `--engine={mujoco,libero_offscreen_render}` argparse arg.
* Drop the `create_simulation` import + the engine-routing block.
  Restore the simpler `Simulation(tool_name="libero_sim", mesh=False)`
  construction that PR #26 had pre-6053bd0.
* Drop the engine-conditional gating around scene-prewarm + camera
  recording (always runs now, since there's only one engine).
* Drop the `engine=...` token from the second grep-stable print line
  (was a one-shot disambiguator added in 6053bd0; redundant now).
* Drop the `--engine=libero_offscreen_render` example from the Usage
  section.
* Rewrite the Verification-status appendix:
  - Removed the 3-row mock/groot/offscreen+in-process measurement
    table (the offscreen row was always misleading per yesterday's
    correction; the mujoco row is now the only one and matches
    `policy=groot` at ~54 s/ep @ ~0.60-0.92).
  - Documents the full upstream catch-up wave: #168 → #172 → #173 →
    #175 → #180 → #184 → #186.
  - Notes the run-to-run variance bounds (0.60 / 0.80 / 0.92 in
    different processes); acceptance is `success_rate > 0`, not a
    specific number.

CHANGES TO `README.md` (root) + `examples/README.md`:

* Drop the `libero_offscreen_render` row from the matrix table.
* Update the `mujoco` row's wall-time + success-rate to ~54 s/ep @
  0.60-0.92 (groot), measured 2026-05-21 against current upstream main.
* Update the footnote to cite the full upstream catch-up wave (#168 →
  #186) and PR #186's reported mean success_rate=0.92.

VERIFIED:

* `--policy=mock --n-episodes 2 --seed 42` → wall=27.4s, success=0.00,
  MP4 written under rollouts/2026_05_21/.
* `--policy=groot --no-auto-server --port 8000 --task libero-10-…SCENE5
  --n-episodes 5 --seed 42` → wall=267.9s, success=0.60 (3/5), MP4
  pair written.
* AST parse + import smoke clean.

DEPENDENCIES: `--policy=groot` numbers reproduce only on a
`strands-robots` install that includes the full upstream catch-up
through #186 (currently merged on upstream main as of 2026-05-21).
The example file still works on `--policy=mock` against any
`strands-robots` install with [sim-mujoco,benchmark-libero] extras.

* feat(libero_mujoco): close ZMQ gap via PR #188 — success_rate=1.00 (5/5)

Validated 2026-05-21 against [`strands-labs/robots#188`](https://github.com/strands-labs/robots/pull/188)
(closes #187, layered on top of all the earlier upstream fixes #168 →
#172 → #173 → #175 → #180 → #184 → #186). PR #188's three coordinated
fixes are:

1. **Spec-driven instruction fallback** — the dominant cause of the
   ZMQ-side gap. `examples/libero_mujoco.py` calls
   `sim.evaluate_benchmark(...)` without `instruction=`, so the empty
   string propagated all the way to the language-conditioned GR00T
   policy as `annotation.human.action.task_description=[""]`. PR #188
   adds `BenchmarkProtocol.instruction` + an `_evaluate_with_spec`
   fallback so spec-supplied instructions reach the policy by default.
2. **Per-episode `policy.reset(seed=)` plumbing** for SERVICE-mode
   reproducibility (mirrors PR #180's in-process `set_eval_seed` flow).
3. **Server-side determinism wrapper** (this commit) — drop-in
   docker-mountable wrapper that sets `cudnn.deterministic=True` +
   `CUBLAS_WORKSPACE_CONFIG=":4096:8"` server-side and applies the
   per-episode seed PR #188's client-side plumbing forwards. Optional;
   only needed for bit-exact reproducibility.

Both seeds 42 and 100 hit `success_rate=1.00 (5/5)` in ~44 s through
this PR's example file (faster than upstream's reported 52 s for seed
42, and decisively beating the 4/5 / 224 s for seed 100):

| seed | success_rate | wall_time |
|------|--------------|-----------|
| 42   | 1.00 (5/5)   | 44.8 s    |
| 100  | 1.00 (5/5)   | 44.3 s    |

Pre-#188 (commit `92d30b3`) the same example produced 0.20-0.60
across runs because the policy was getting empty instructions.

CHANGES:

* `examples/libero_mujoco.py` Verification-status appendix updated
  with the post-#188 numbers (5/5 / ~9 s/ep) + chronicle of the
  upstream catch-up wave including #188 + a section on the optional
  server-side determinism wrapper.
* `README.md` (root) backend-matrix `mujoco` row updated to ~9 s/ep
  @ 1.00 (was ~54 s/ep @ 0.60-0.92). Footnote cites #188 as the
  unblocker that closed the ZMQ-side gap.
* `examples/README.md` matches root README.
* `examples/gr00t_server_deterministic_wrapper.py` — drop-in
  docker-mountable wrapper for users who need bit-exact CUDA
  determinism. The example file's docstring references it but works
  WITHOUT it (verified at 5/5 above). Set
  `STRANDS_GR00T_SERVER_SEED=<int>` to override the default seed.

VERIFIED:
* `examples/libero_mujoco.py --policy groot --port 8000 --task libero-10-…SCENE5
  --n-episodes 5 --seed 42` → 5/5 in 44.8s (auto-server, no wrapper)
* Same with `--seed 100` → 5/5 in 44.3s
* AST parse + import smoke clean

DEPENDENCIES: `--policy=groot` numbers reproduce only on a
`strands-robots` install that includes the full upstream catch-up
through #188. The example file still works on `--policy=mock` against
any `strands-robots` install with [sim-mujoco,benchmark-libero] extras.

* refactor(examples): move LIBERO scripts to examples/libero/ + fix data_config bug

Addresses sundargthb's review feedback on PR #26:

1. **Subfolder reorganization** (#26 review). Move LIBERO-specific
   scripts under `examples/libero/` so the top-level `examples/` doesn't
   get crowded as R8/R12/R23 PRs add their own backend examples:

       examples/libero_mujoco.py                       → examples/libero/run_mujoco.py
       examples/libero_mujoco_agent.py                 → examples/libero/run_mujoco_agent.py
       examples/gr00t_server_deterministic_wrapper.py  → examples/libero/gr00t_server_deterministic_wrapper.py

   IMPORTANT: kept the `run_` prefix instead of the originally-suggested
   bare `mujoco.py` / `mujoco_agent.py` names because Python's
   sys.path[0]-prepend behaviour means `python examples/libero/mujoco.py`
   would shadow the installed PyPI `mujoco` package. The shadow caused
   `mujoco.MjSpec()` to fail deep inside `strands_robots` since
   `import mujoco` resolved to our example file. `run_mujoco.py` keeps
   the subfolder grouping sundargthb wanted while sidestepping the
   import collision. Sibling files inside `libero/` cross-reference
   each other by relative module name (`run_mujoco.py` ↔
   `run_mujoco_agent.py`) so future R8/R12 examples can sit next to
   them under `libero/run_isaac.py` etc. without colliding with
   `isaac` or other PyPI package names.

2. **`data_config` mismatch fix** (sundargthb's inline comment on
   line 123 of the agent file). The agent prompt specified
   `data_config='libero'` while the programmatic file uses
   `data_config='libero_panda'`. Only `libero_panda` is in
   `DATA_CONFIG_MAP`, so the agent flow would KeyError at policy
   construction with no useful hint — the LLM passed exactly what we
   told it to. Fixed both occurrences in `run_mujoco_agent.py`
   (lines 114 and 123) so the agent file matches the programmatic
   file.

3. **All cross-references updated** — README.md (root) backend matrix,
   examples/README.md table + section pointers, examples/MIGRATION.md
   markdown link URLs, in-file docstring usage examples + sibling
   refs, and the wrapper script's own mount-path doc.

VERIFIED:
* `python examples/libero/run_mujoco.py --policy mock --n-episodes 2 --seed 42`
  → wall=27.6s, success=0.00, MP4 written.
* AST parse + import smoke clean for all three moved files.
* `mujoco.MjSpec()` no longer collides — confirmed by the smoke
  passing where it previously failed.

The post-#188 success_rate=1.00 (5/5) measurement on
`libero-10-LIVING_ROOM_SCENE5_…` (verified yesterday on commit
`a480b15`) carries forward unchanged — the move is path-only, the
example contents are byte-identical aside from docstring path
references.

* docs(examples): trim historical PR-link chains per cagataycali review

Removed the upstream catch-up wave PR chain (#168, #172, #173, #175,
#180, #184, #186, #188) from the verification sections of both
examples/README.md and examples/libero/run_mujoco.py docstring.

Kept the practical reproduction info: measured numbers, hardware,
seeds, suite/task, optional determinism-wrapper section. Forward-
pointing nav (R8/#15, R12/#19, etc.) and operational hints
(`#147`, `#148`, `#165`) preserved — different in nature from
the historical chain cagataycali flagged.

Addresses:
- examples/README.md L90 review comment
- examples/libero/run_mujoco.py L70 review comment

* fix(libero/agent): make --policy=groot work end-to-end

The agent file shipped broken on its advertised --policy=groot mode.
Symptom: 5 distinct failures stacked, eval never produced an MP4 with
real rollout content.

Root cause: too much was delegated to the LLM. The agent thrashed
through 25 tool calls trying to pick the right action from
Simulation's 64-action enum, kept verb-matching 'run' -> run_policy
or 'eval' -> eval_policy (both skip the LIBERO observation adapter,
trigger 'State key state.x must be in observation' server-side), and
along the way redownloaded a cached checkpoint, spawned a duplicate
container ignoring --port, and picked LeRobot Dataset recorder by
mistake (which crashed on the existing rollouts/ dir).

Fix: move deterministic plumbing into the script, leave the LLM-
suited bits to the agent.

Script now owns:
- GR00T container lifecycle (start, wait-for-load, teardown) via the
  same gr00t_inference(action='lifecycle', ...) block as run_mujoco.py
- LIBERO scene pre-warm (spec._generate_scene_from_bddl ->
  sim.load_scene -> spec.prewarm). Without this, GR00T server rejects
  the first observation with 'Video key video.image must be in
  observation' because cameras aren't registered yet.
- MP4 recording via start_cameras_recording / stop_cameras_recording
  (NOT LeRobot Dataset)
- Default-task fallback for the aspirational placeholder
  libero-spatial-pick_up_the_red_cube
- Argparse expanded to mirror run_mujoco.py: --auto-server /
  --no-auto-server, --image, --container, --checkpoint-dir

Agent now owns: one tool call (evaluate_benchmark with kwargs from
prompt context) + the natural-language summary. Prompt explicitly
names the action so the LLM can't verb-match its way to a sibling
sub-tool.

Validation on L4 / Docker dev box, libero-10/SCENE5, seed 42:
- --policy=groot, 5 eps: success_rate=1.00 (5/5), wall=154.2s, MP4
  1.3 MB image / 2.7 MB wrist (matches run_mujoco.py's 1.7 MB / 3.4 MB
  proportionally, eval portion is the same; agent overhead ~100s for
  LLM round-trips)
- --policy=mock, 2 eps: success_rate=0.0 (expected), wall=155.4s, MP4
  written

Filename now includes --agent suffix
(--policy=groot--agent__image.mp4) so post-hoc analysis can tell
agent-driven from programmatic rollouts at a glance.

Out of scope (filed separately if needed): an upstream
asyncio.run() layering issue in strands-labs/robots
_async_utils._resolve_coroutine — the executor fallback at line 29
papers over it for evaluate_benchmark, but it surfaces noisily in
tracebacks and would benefit from a proper fix in that repo.

* fix(libero/agent): wire synchronous recording (closes upstream #191)

Upstream strands-labs/robots#192 (merged at 974a8f6) added the
synchronous-recording API I filed in #191:
- evaluate_benchmark now accepts on_frame=
- start_cameras_recording_synchronous returns (on_frame, finalize)
  closures instead of spawning a daemon recorder thread

Wire the agent file to the new path so its rollout videos no longer
suffer from the daemon-thread / mjData race that produced 4-second
truncated MP4s with greenish GL clear-colour artifacts under
multi-threaded eval (Strands Agent worker thread + daemon recorder).

Shape: @tool wrapper inside main() that captures sim/video_dir/
rec_name and on_frame_cb from the outer scope. Agent picks the
wrapper, fills its kwargs from natural language, returns. Two extra
bits the upstream docstring's example glosses over but matter in
practice:

1. Explicit width=640, height=480 on
   start_cameras_recording_synchronous. Without these the eval
   thread's renderer cache produces variable-shape arrays in the
   buffer that crash imageio's FFmpeg encoder mid-stream with
   BrokenPipeError + glibc malloc_consolidate noise.

2. finalize_cb invoked from the script main thread (in a try/finally
   around the agent call), NOT from inside the @tool wrapper. The
   wrapper runs on the Strands worker thread; FFmpeg subprocess.Popen
   spawned from a non-main thread also crashes with BrokenPipeError.
   Moving the finalize call to main resolves it.

Validation on L4 / Docker dev box, libero-10/SCENE5:

| Run | wall | success_rate | MP4 |
|---|---|---|---|
| --policy=groot, seed=42, 5 eps | 338 s | 1.00 (5/5) | 5.9 MB image / 13 MB wrist; 1135 frames |
| --policy=groot, seed=42, 1 ep  |  75 s | 1.00 (1/1) | 1.2 MB image / 2.4 MB wrist;  227 frames |
| --policy=mock,  seed=42, 2 eps |  85 s | 0.00 (expected) | 255 KB default |

The 1135 frames at 5 eps = exactly 1 frame per simulation step
(5 × ~227 steps), matching evaluate_benchmark's per-step on_frame
hook. Compare to the daemon-thread version (cf. PR #26 commit
66005d4) which captured 120 frames in 4 seconds for the same 5-ep
eval — a ~10x improvement in capture rate plus boundary frames are
now clean (no greenish artifact on episode reset).

Sample frames extracted at steps 30 / 100 / 200 / 230 (episode 0->1
boundary) all show the expected scene state with no rendering
artifacts.

Slight wall-time increase vs daemon-thread shape (175 -> 338 s for
5 eps) is the per-step render cost paying for 100% capture instead
of ~2-3% — acceptable trade for matrix-quality video output.

* refactor(libero/agent): drop @tool wrapper, agent picks evaluate_benchmark from Simulation surface

Earlier (b83257d) the agent file wrapped evaluate_benchmark in a
custom @tool to thread upstream PR #192's synchronous recorder
on_frame closure past Strands' tool-dispatch JSON boundary. That
guaranteed bit-exact 1-frame-per-step videos but at the cost of:
- 5-6x wall-time multiplier (per-step render on eval thread)
- ~70 lines of closure-plumbing
- agent picks 1-of-1 wrapper instead of 1-of-64 Simulation actions
  — the 'agent picks the right tool' demo aspect was degenerate

Reverting to daemon-thread start_cameras_recording + Agent(tools=[
sim]) lets the agent pick evaluate_benchmark from the full Simulation
action enum via natural language. The prompt explicitly names the
action so the LLM doesn't verb-match 'run' -> run_policy or 'eval' ->
eval_policy (both skip the LIBERO observation adapter).

Trade-off captured in the docstring: the daemon-thread recorder
races with the eval thread on shared mjData under multi-threaded
Agent dispatch, so frame coverage is ~20% of sim steps and an
occasional greenish GL clear-colour artifact may appear. Users who
need guaranteed-clean video should use run_mujoco.py programmatically;
the agent file's job is the natural-language demo.

Validation on L4 / Docker, libero-10/SCENE5, seed 42:

| Run | wall | success_rate | image MP4 / wrist | frames |
|---|---|---|---|---|
| --policy=groot, 5 eps | 61.3s | 1.00 (5/5) | 1.8 MB / 3.5 MB | 224 / 221 |
| --policy=mock, 2 eps  | 36.6s | 0.00 (expected) | 219 KB | — |

Compared to b83257d's wrapper version on the same task/seed:
- 5.5x faster wall-time (61s vs 336s)
- 5x fewer frames (224 vs 1135) — daemon thread starved of GIL
- Sample frames at steps 30/100/200/280 verified clean (no greenish
  artifacts) on this run; race risk documented but not triggered

Net diff: -69 lines / +57 lines (line count parity with run_mujoco.py
at 421 / 424 lines).

* review: address @cagataycali's CHANGES_REQUESTED on PR #26

P1 (blocking, doc-drift that breaks copy-paste users):

1. MIGRATION.md L41/L96/L168: s/data_config="libero"/"libero_panda"/.
   Bare 'libero' raises KeyError at policy construction (DATA_CONFIG_MAP
   only registers libero_panda). Same bug sundargthb caught in the
   agent file via cf7f8c6 — was missed in the migration doc.

2. MIGRATION.md L86/L163/L184: s/add_robot("panda"/add_robot("robot"/
   plus s/robot_name="panda"/omit/. LIBERO scene MJCFs name their
   Panda 'robot' (RoboSuite convention), so picking it here keeps the
   resolved robot stable across evaluate_benchmark's on_episode_start
   scene reload — same rationale captured in run_mujoco.py:297-302.

3. MIGRATION.md L42 narrative: `Agent(tools=[sim, gr00t_inference])`
   -> `Agent(tools=[sim])` and updated the table-cell prose to reflect
   the lifecycle-vs-agent split that's actually shipped — script owns
   container lifecycle / scene pre-warm / MP4 recording, agent only
   invokes evaluate_benchmark.

P2 (fold-in before merge):

4. run_mujoco.py L354: removed unused `import random as _random`
   (leftover from pre-simplification of scene pre-warm).

5. pyproject.toml: extended hatch lint targets to `examples/`. Surfaced:
   - 1x F401 unused import (item 4, fixed)
   - 4x E402 in deterministic wrapper (intentional ordering)
   - 1x E501 in deterministic wrapper L88 (123 chars)
   - black/isort reformats across all three example files
   All resolved; E402 sites annotated with `# noqa: E402  # <reason>`.

P3 (followups, addressed the quick wins):

6. __init__.py: switched `__version__ = "0.2.0"` (hardcoded literal)
   to `importlib.metadata.version("strands-robots-sim")` with a
   PackageNotFoundError fallback. Single source of truth: the VCS tag.

7. __init__.py: dropped `warnings.warn(message, DeprecationWarning)`
   before `raise ImportError`. Under `-W error::DeprecationWarning`
   the warning would mask the actionable ImportError.

8. gr00t_server_deterministic_wrapper.py: noqa-annotated 4 E402 sites
   and wrapped the 123-char print line.

9. README.md: added a footnote to the `~9 s/ep @ 1.00` matrix cell
   noting it's a single-sample on L4 with the variance history.

P3.10 (recorder warmup noise) deferred — follow-up to gate retries on
cameras_ready rather than wall-clock budget.

P4 (style):

- run_mujoco_agent.py: deleted the dead 'Optional follow-up' comment-
  with-code block at end of file.

Validation:

- black --check / isort --check / flake8 (max-line-length=120): all
  clean across strands_robots_sim/ + examples/
- AST parse on all four touched .py files: ok
- import strands_robots_sim; legacy SimEnv import: clean ImportError
- run_mujoco.py --policy mock --n-episodes 2 --seed 42: success_rate=0.00
  (expected mock), wall_time=26.9s, MP4 written

Net diff: 7 files, +75/-95 lines.

---------

Co-authored-by: opencode <opencode@local>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

2 participants