refactor(compute): unify gateway restart reconciliation (#2743)

* refactor(compute): unify gateway restart reconciliation

Remove the Docker-specific gateway shutdown cleanup and reconcile persisted running intent through ComputeDriver::StartSandbox for Docker, Podman, and VM drivers. Explicitly stopped sandboxes remain stopped.

Refs #2417

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* refactor(compute): stop local sandboxes on shutdown

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* test(e2e): match managed Podman containers

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

* fix(compute): synchronize lifecycle sweeps

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

---------

Signed-off-by: Drew Newberry <anewberry@nvidia.com>

Author: drew

.agents/skills/debug-openshell-cluster/SKILL.md

@@ -194,6 +194,13 @@ For source checkout development, restart the local gateway with:
 mise run gateway:docker
 ```
 
+During a graceful gateway restart, Docker, Podman, and VM sandboxes with
+running intent should stop before the gateway exits and restart after it
+returns. Check for `Stopped sandbox during gateway shutdown` and `Started
+sandbox during gateway startup` in gateway logs. A sandbox explicitly stopped
+through the CLI remains stopped. Kubernetes sandboxes are cluster-owned and do
+not follow this local gateway lifecycle.
+
 ### Step 5: Check Podman-Backed Gateways
 
 ```bash

architecture/compute-runtimes.md

@@ -110,6 +110,15 @@ compute to zero, and VM retains its launch request and writable overlay beside
 a stop marker. Delete remains a separate operation that removes these
 resources.
 
+On graceful gateway shutdown, persisted running intent for Docker, Podman, and
+VM is stopped through the shared `StopSandbox` RPC before any gateway-managed
+driver process exits. The gateway does not persist `Stopped` for this
+infrastructure event. On startup, it reconciles the retained intent through the
+shared idempotent `StartSandbox` RPC before watch processing begins. Explicitly
+`Stopped` sandboxes are excluded from both sweeps. Kubernetes workloads are
+cluster-owned and continue running without gateway shutdown or startup
+lifecycle calls.
+
 ## Deletion Lifecycle
 
 Lifecycle requests use per-sandbox gates to serialize stop, start, and

crates/openshell-driver-docker/README.md

@@ -26,6 +26,11 @@ policy. Start starts that same container, so files in the resolved OCI
 workspace remain available. A durably stopped sandbox is excluded from
 gateway startup recovery and stays stopped across gateway restarts. Delete
 continues to force-remove the container and clean up driver-owned material.
+Graceful gateway shutdown sends `StopSandbox` for each sandbox whose persisted
+phase requires running compute without changing that persisted intent. On
+startup, the gateway sends an idempotent `StartSandbox` request for the same
+sandboxes, restarting their retained containers. Explicitly stopped sandboxes
+remain excluded.
 
 Before creating the container, the driver inspects the final sandbox image and
 captures its immutable image ID, raw OCI `Config.User`, and OCI

crates/openshell-driver-docker/src/lib.rs

@@ -1055,69 +1055,6 @@ impl DockerComputeDriver {
         }
     }
 
-    pub async fn stop_managed_containers_on_shutdown(&self) -> Result<usize, Status> {
-        let containers = self.list_managed_container_summaries().await?;
-        let targets = containers
-            .into_iter()
-            .filter_map(|container| {
-                let state = container.state.unwrap_or(ContainerSummaryStateEnum::EMPTY);
-                if container_state_needs_shutdown_stop(state) {
-                    summary_container_target(&container)
-                } else {
-                    None
-                }
-            })
-            .collect::<Vec<_>>();
-        let target_count = targets.len();
-        let mut stopped = 0usize;
-        let mut failures = Vec::new();
-        let stop_timeout_secs = self.config.stop_timeout_secs;
-
-        let mut stop_results = futures::stream::iter(targets.into_iter().map(|target| {
-            let docker = self.docker.clone();
-            async move {
-                let result = docker
-                    .stop_container(
-                        &target,
-                        Some(
-                            StopContainerOptionsBuilder::default()
-                                .t(docker_stop_timeout_secs(stop_timeout_secs))
-                                .build(),
-                        ),
-                    )
-                    .await;
-                (target, result)
-            }
-        }))
-        .buffer_unordered(16);
-
-        while let Some((target, result)) = stop_results.next().await {
-            match result {
-                Ok(()) => {
-                    stopped += 1;
-                }
-                Err(err) if is_not_found_error(&err) || is_not_modified_error(&err) => {}
-                Err(err) => {
-                    warn!(
-                        container = %target,
-                        error = %err,
-                        "Failed to stop Docker sandbox container during shutdown"
-                    );
-                    failures.push(target);
-                }
-            }
-        }
-
-        if !failures.is_empty() {
-            return Err(Status::internal(format!(
-                "failed to stop {} of {target_count} Docker sandbox containers during shutdown",
-                failures.len()
-            )));
-        }
-
-        Ok(stopped)
-    }
-
     async fn reserve_pending_sandbox(&self, sandbox: &DriverSandbox) -> Result<(), Status> {
         let mut pending = self.pending.lock().await;
         if pending
@@ -3200,15 +3137,6 @@ fn summary_container_target(summary: &ContainerSummary) -> Option<String> {
         .or_else(|| summary_container_name(summary))
 }
 
-fn container_state_needs_shutdown_stop(state: ContainerSummaryStateEnum) -> bool {
-    matches!(
-        state,
-        ContainerSummaryStateEnum::RUNNING
-            | ContainerSummaryStateEnum::RESTARTING
-            | ContainerSummaryStateEnum::PAUSED
-    )
-}
-
 /// States from which a managed container can be brought back to running by
 /// `start_container`. Skip `Restarting` (already coming up), `Removing`,
 /// `Dead` (terminal), `Paused` (needs `unpause`, not `start`), and

crates/openshell-driver-podman/README.md

@@ -32,6 +32,12 @@ The stop call waits until Podman reports the container as stopped or exited.
 This keeps an immediate start from racing a rootless Podman stop that is still
 finishing after its API request returns.
 
+Graceful gateway shutdown sends `StopSandbox` for each sandbox whose persisted
+phase requires running compute without changing that persisted intent. On
+startup, the gateway sends an idempotent `StartSandbox` request for the same
+sandboxes, restarting their retained containers. Explicitly stopped sandboxes
+remain excluded.
+
 ## Architecture
 
 The Podman driver communicates with the Podman daemon over a Unix socket and

crates/openshell-driver-vm/README.md

@@ -194,10 +194,11 @@ during the first prepare.
 
 The driver also writes the accepted `DriverSandbox` launch request to
 `<state-dir>/sandboxes/<id>/sandbox.pb`. If the gateway restarts, it starts a
-new VM driver process; that process scans the sandbox state directories,
-restarts each persisted VM launcher, and preserves any existing `overlay.ext4`
-instead of cloning a fresh overlay template. If a restart happened before the
-overlay was created, the driver creates it during the start attempt.
+new VM driver process. During graceful shutdown, the gateway first sends the
+shared `StopSandbox` request for each persisted running-intent sandbox, which
+stops its launcher while retaining the launch request and `overlay.ext4`.
+After driver initialization, the gateway sends the idempotent `StartSandbox`
+request for that retained intent. Explicitly stopped sandboxes remain excluded.
 
 Stop writes a marker in the sandbox state directory before terminating
 the launcher and releasing host GPU and network allocations. It retains