Make native reproduction environments survive agent tool calls

dev/repro_env/README.md

@@ -0,0 +1,72 @@
+# Prepared reproduction environment
+
+CI starts the product server, runner and mock model server in a persistent
+sandbox before repro-agent launches. It configures both real native CLIs with
+mock providers and isolated product/CLI state. The workflow owns their lifetime;
+they survive the agent CLI disconnecting and individual shell calls ending.
+
+Run each journey in the foreground through the connection wrapper:
+
+```sh
+python -m dev.repro_env exec -- python -m pytest \
+  tests/e2e_ui/messages/test_native_claude_render_parity.py::test_native_claude_message_render_parity \
+  --ui-skip-build --video=on --output=recordings/native-claude
+```
+
+Use `native_codex_mock_session` or `native_claude_mock_session` in authored UI
+tests. Existing fixtures attach to the prepared server and runner, and
+`mock_llm_server_url` addresses its model server. They do not provision another
+runner or decide the model backend from ambient credentials.
+
+Attachment supports HTTP/browser journeys and native session fixtures. Tests
+that directly kill/restart a server or runner or access the fixture database
+require their own environment; run those outside `dev.repro_env exec`. Missing
+process/database state produces an explicit error. The three connection
+variables must be supplied together; use the wrapper rather than setting only
+one of them.
+
+Standalone native mock fixtures save an existing provider config to an
+owner-only `config.yaml.e2e-backup` before replacing it and restore it on exit.
+If the test process is killed, recover that backup before retrying; subsequent
+runs refuse to overwrite it.
+
+Arbitrary Python/Playwright commands also work with the wrapper. They receive
+`OMNIGENT_REPRO_SERVER_URL`, `OMNIGENT_REPRO_MODEL_URL`, and
+`OMNIGENT_REPRO_RUNNER_ID`. These URLs are valid only inside that invocation;
+reuse session IDs across invocations, not the temporary URLs. HTTP, SSE and
+terminal WebSockets all use the same product endpoints. Put shared files in the
+worktree, since `/tmp` is private to each sandbox.
+
+Script model responses using the existing helpers in `tests/e2e_ui/conftest.py`
+(`configure_mock_llm`, `set_fallback_mock_llm`, `reset_mock_llm`). Responses may
+include text, tool calls, delays, errors or streaming interruptions. Session
+creation and launch options remain the regular product API. Mock state is shared
+within one reproduction attempt; configure it before each journey and run
+journeys sequentially.
+
+Record before performing the reported actions and close the browser context to
+finalize video, including on assertion failure. A crash, missing reply, or stuck
+approval can be reproduction evidence. A successful canned reply is only a
+connectivity check; the authored test decides whether the reported bug occurred.
+These mocks validate native integration, not live-provider/model behavior.
+
+`python -m dev.repro_env status` prints startup status. Inspect
+`.omnigent/repro-env/` for process logs, product logs under `data/`, provider
+configuration, database and model request statistics. Connection failures must
+be reported with their actual diagnostics; do not substitute callbacks or fake
+terminal output and claim a real native turn.
+
+Shutdown also saves the mock's captured request bodies. These cover requests
+since the last `reset_mock_llm`; save them before resetting if an earlier
+journey's model traffic is needed as evidence.
+
+The workflow stops the environment after session completion and recording
+normalization, then bundles diagnostics even on failure. A six-hour lease bounds
+its lifetime if normal cleanup cannot run. `serve` is a foreground supervisor
+intended for the workflow's persistent sandbox; starting it as a background job
+in an agent shell does not give it that lifetime.
+
+Each `serve` attempt requires a fresh output directory with mode 0700. Preserve
+the previous directory for diagnostics and select another with `--output PATH`
+(before the `serve` subcommand). Startup never clears a stop request: the
+workflow may already have requested cancellation before the supervisor starts.

dev/repro_env/__init__.py

@@ -0,0 +1 @@
+"""Workflow-owned reproduction runtime and connections for sandboxed clients."""

dev/repro_env/__main__.py

@@ -0,0 +1,71 @@
+"""Use `python -m dev.repro_env exec -- <command>` inside each agent shell."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import subprocess
+from pathlib import Path
+
+import httpx
+
+from .runtime import serve
+from .transport import Relay
+
+
+def execute(output: Path, command: list[str]) -> int:
+    state = json.loads((output / "environment.json").read_text())
+    if state["status"] != "ready":
+        raise RuntimeError(f"Reproduction environment is {state['status']}; inspect {output}")
+    with (
+        Relay(unix_target=output / "server.sock") as server,
+        Relay(unix_target=output / "model.sock") as model,
+    ):
+        env = dict(os.environ)
+        env.update(
+            OMNIGENT_REPRO_SERVER_URL=f"http://127.0.0.1:{server.port}",
+            OMNIGENT_REPRO_MODEL_URL=f"http://127.0.0.1:{model.port}",
+            OMNIGENT_REPRO_RUNNER_ID=state["runner_id"],
+        )
+        for key in ("NO_PROXY", "no_proxy"):
+            env[key] = ",".join(filter(None, (env.get(key), "localhost,127.0.0.1,::1")))
+        with httpx.Client(trust_env=False, timeout=5) as client:
+            client.get(f"{env['OMNIGENT_REPRO_MODEL_URL']}/stats").raise_for_status()
+            status = client.get(
+                f"{env['OMNIGENT_REPRO_SERVER_URL']}/v1/runners/{state['runner_id']}/status"
+            )
+            status.raise_for_status()
+            if not status.json().get("online"):
+                raise RuntimeError(f"Reproduction runner is offline; inspect {output}")
+        return subprocess.call(command, env=env)
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--output", type=Path, default=Path(".omnigent/repro-env"))
+    commands = parser.add_subparsers(dest="action", required=True)
+    start = commands.add_parser("serve")
+    start.add_argument("--lease-seconds", type=int, default=21600)
+    run = commands.add_parser("exec")
+    run.add_argument("command", nargs=argparse.REMAINDER)
+    commands.add_parser("stop")
+    commands.add_parser("status")
+    args = parser.parse_args()
+    output = args.output.resolve()
+    if args.action == "serve":
+        return serve(output, args.lease_seconds)
+    if args.action == "stop":
+        (output / "stop").touch()
+        return 0
+    if args.action == "status":
+        print((output / "environment.json").read_text())
+        return 0
+    command = args.command[1:] if args.command[:1] == ["--"] else args.command
+    if not command:
+        parser.error("exec requires a command after --")
+    return execute(output, command)
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())