Motivation
CS Bridge already has session memory: when a job hits walltime, the expired session is flagged and Restart resubmits with the same partition, account, and resources. Files on the shared filesystem survive because we never touched them.
But everything live is lost — the running processes, the in-flight computation, the open terminals, the language servers, the editor's working state. A researcher who was mid-run comes back to a clean login and restarts from whatever on-disk checkpoint their application happened to leave behind, if any.
We want suspend/resume for a CS Bridge session: when a job is interrupted by a timeout, its running processes are captured and restored on the next allocation — the window comes back feeling like the researcher minimized it and refocused, not like they reconnected to a fresh node. This moves us from "your files are safe" to "your session is safe."
User story
I'm working in a CS Bridge session with a scientific job running and a couple of terminals open. The walltime expires. I click Resume. The new allocation comes up, my computation is still running and making progress, my terminals are reattached, and my editor is where I left it. I didn't lose my place.
Scope the first pass to the expected interruption — walltime expiry — which is the common case and the one we control: we know roughly when it's coming, and SLURM gives us a grace window before the kill. Unexpected node failure is a harder variant.
Scope
Roughly descending in importance and difficulty:
- Running process trees — foreground and background processes, including a long computation, resumed mid-flight with memory state intact (not restarted from zero).
- Open files / file descriptors into the shared filesystem and scratch.
- Network connections — needs a careful definition. The Dev Tunnel and SSH server are our transport and will be re-created on the new node; that's ours to rebuild, not the user's state to preserve. Sockets the user's processes hold to external services are likely dead after the gap. Part of this spike is deciding what "preserve connections" can honestly mean.
- Terminal sessions — the shells the user had open, reattached rather than respawned.
- VS Code editor state — open editors, unsaved buffers, layout, running tasks — reconnected so it reads as "refocus," not "fresh connect."
These are not equally expensive. A v1 that nails 1, 2, 4, and 5 while being honest about 3 is a real win.
Acceptance Criteria
- A long-running computation keeps progressing across a walltime expiry + Resume, rather than restarting.
- Open terminals come back attached to their original shells.
- VS Code reconnects under a stable session identity and reads as a refocus, not a fresh connect.
- The mechanism works for an unprivileged user on a real target cluster, with no root and no custom kernel modules.
- The feature is honest about what it doesn't preserve (external sockets, GPU state) and fails safe to the existing clean-restart path when restore isn't possible.
- Checkpoint images are cleaned up after a successful restore or a fall-through to clean restart, so stale multi-GB dumps don't accumulate in scratch.
Additional Context
Where does this live in the architecture?
The work splits across the two repos along the existing seam.
linkspan — the on-node mechanism. linkspan already runs on the compute node and owns the local environment: a workflow engine, a REST API under /api/v1/, an internal/process/ manager for background processes, and a subsystems/ tree (mount/, vscode/, tunnel/, jupyter/). A checkpoint/restore capability fits as a new subsystems/checkpoint/, exposed as REST endpoints (POST /api/v1/checkpoint, /restore) and/or workflow actions (checkpoint.create, checkpoint.restore) in the same style as the existing actions — so the restore step wires into the new allocation's startup workflow the way tunnel.devtunnel_create does today. The existing internal/process/ manager is the obvious starting point for "what is running," though the hard part is everything it doesn't track (memory, fds, child trees spawned outside linkspan).
cs-bridge — orchestration and UX. cs-bridge already drives the SLURM lifecycle and opens vscode-remote://ssh-remote+cshost-<sessionId>/{HOME}. The additions: trigger a pre-emptive checkpoint via linkspan before the job is killed (it already polls sacct, so it knows when walltime is near); persist a checkpoint handle alongside session metadata in ~/.cybershuttle/sessions.json; and on resume, submit a new job whose linkspan workflow restores, keeping the same sessionId so the vscode-remote:// URI is stable and VS Code reconnects to "the same" host while the new node, tunnel, and relayed SSH endpoint are remapped underneath. This probably wants to be a distinct Resume action rather than overloading Restart (which intentionally means "fresh job, same resources") — restore can fail, and clean restart is the fallback.
The non-trivial parts
None of these have an obvious answer, and the right strategy probably combines several mechanisms.
- Unprivileged checkpoint/restore on HPC. The defining constraint: users on shared clusters can't assume root, kernel modules, or capabilities. What actually works across heterogeneous clusters? DMTCP (userspace,
LD_PRELOAD, long HPC track record) is the most realistic starting point; CRIU is more powerful but typically wants CAP_SYS_ADMIN — investigate how far rootless gets on real cluster kernels; container-scoped C/R (runc/Podman/Apptainer) is more tractable but assumes the session runs in a container; app-level checkpointing is the blessed HPC approach but needs the user's code to cooperate, so it's at best a fast path, not the general mechanism. A hybrid is plausible.
- The grace window. SLURM can deliver a warning signal before
SIGKILL (the --signal flag and its grace period; also --requeue). Is the grace enough to dump potentially multi-GB memory images, and how do we wire signal handling SLURM → linkspan → checkpoint? Dumping state isn't free and the clock is the enemy: at parallel-FS write bandwidth on the order of GB/s, a few-GB image is seconds, but a large one can exceed a short grace period — so image size, write target, and the configured grace need to be reasoned about together, not separately.
- Where checkpoint state lives. Memory images can be large and must survive node teardown — shared/parallel FS, scratch, or burst buffer — which raises I/O cost (again against the grace clock), retention, and cleanup.
- GPU work. GPU checkpointing is a known hard problem (driver/runtime/device-memory state). NVIDIA's
cuda-checkpoint + CRIU is the thread to pull, but it's reasonable to scope GPU jobs out of v1 and detect-and-warn rather than silently fail.
- VS Code continuity. Making reconnection feel like "refocus" is partly orthogonal to process C/R. Wrapping session shells in
tmux/dtach so terminals survive and re-attach is likely cheaper and more robust than C/R-ing TTYs, and may carry most of the perceived seamlessness. Unsaved buffers likely come free from VS Code's hot-exit once sessionId is stable; the open question is whether a respawned extension host is acceptable.
- What's actually restorable. A clear-eyed taxonomy — full fidelity (process trees), approximate (terminals via re-attach), genuinely lost (live external sockets) — so the feature sets honest expectations instead of promising perfection.
Bonus: Resume onto a different partition or resource configuration
Once same-config resume works, the same machinery enables migration: checkpoint on one resource and restore on another — e.g. a researcher develops on a small interactive node, then moves the session onto a high-performance/GPU partition without losing running processes, and back again.
This is strictly a second milestone — it adds heterogeneity constraints that don't exist in the same-config case and shouldn't block the base: CPU/microarchitecture differences can make a checkpoint non-restorable on the target; moving a process holding GPU state is the hard GPU-C/R problem again; glibc/kernel/library skew between partitions can break restore; and identity (IP, hostname, namespaces) all remaps, though the stable-sessionId trick carries over. Build it only once the base is solid, and expect to constrain it to compatible partition pairs at first.
Reference Material
In-repo: cs-bridge "How It Works" and CONTRIBUTING.md (session model, sessions.json, per-session SSH config); linkspan README (workflow engine, REST API, internal/process/, subsystems/ layout).
External: DMTCP · CRIU · NVIDIA cuda-checkpoint · SLURM --signal/grace/--requeue · runc/Podman/Apptainer · tmux/dtach.
Suggested start: checkpoint and restore a single CPU-bound process under DMTCP across a real SLURM walltime, driven by a linkspan workflow action and triggered by cs-bridge's existing sacct polling — enough to validate the spine before terminals, editor state, or migration. The two gating questions are unprivileged C/R and the grace window; everything else follows.
Motivation
CS Bridge already has session memory: when a job hits walltime, the expired session is flagged and Restart resubmits with the same partition, account, and resources. Files on the shared filesystem survive because we never touched them.
But everything live is lost — the running processes, the in-flight computation, the open terminals, the language servers, the editor's working state. A researcher who was mid-run comes back to a clean login and restarts from whatever on-disk checkpoint their application happened to leave behind, if any.
We want suspend/resume for a CS Bridge session: when a job is interrupted by a timeout, its running processes are captured and restored on the next allocation — the window comes back feeling like the researcher minimized it and refocused, not like they reconnected to a fresh node. This moves us from "your files are safe" to "your session is safe."
User story
Scope the first pass to the expected interruption — walltime expiry — which is the common case and the one we control: we know roughly when it's coming, and SLURM gives us a grace window before the kill. Unexpected node failure is a harder variant.
Scope
Roughly descending in importance and difficulty:
These are not equally expensive. A v1 that nails 1, 2, 4, and 5 while being honest about 3 is a real win.
Acceptance Criteria
Additional Context
Where does this live in the architecture?
The work splits across the two repos along the existing seam.
linkspan— the on-node mechanism. linkspan already runs on the compute node and owns the local environment: a workflow engine, a REST API under/api/v1/, aninternal/process/manager for background processes, and asubsystems/tree (mount/,vscode/,tunnel/,jupyter/). A checkpoint/restore capability fits as a newsubsystems/checkpoint/, exposed as REST endpoints (POST /api/v1/checkpoint,/restore) and/or workflow actions (checkpoint.create,checkpoint.restore) in the same style as the existing actions — so the restore step wires into the new allocation's startup workflow the waytunnel.devtunnel_createdoes today. The existinginternal/process/manager is the obvious starting point for "what is running," though the hard part is everything it doesn't track (memory, fds, child trees spawned outside linkspan).cs-bridge— orchestration and UX. cs-bridge already drives the SLURM lifecycle and opensvscode-remote://ssh-remote+cshost-<sessionId>/{HOME}. The additions: trigger a pre-emptive checkpoint via linkspan before the job is killed (it already pollssacct, so it knows when walltime is near); persist a checkpoint handle alongside session metadata in~/.cybershuttle/sessions.json; and on resume, submit a new job whose linkspan workflow restores, keeping the samesessionIdso thevscode-remote://URI is stable and VS Code reconnects to "the same" host while the new node, tunnel, and relayed SSH endpoint are remapped underneath. This probably wants to be a distinct Resume action rather than overloading Restart (which intentionally means "fresh job, same resources") — restore can fail, and clean restart is the fallback.The non-trivial parts
None of these have an obvious answer, and the right strategy probably combines several mechanisms.
LD_PRELOAD, long HPC track record) is the most realistic starting point; CRIU is more powerful but typically wantsCAP_SYS_ADMIN— investigate how far rootless gets on real cluster kernels; container-scoped C/R (runc/Podman/Apptainer) is more tractable but assumes the session runs in a container; app-level checkpointing is the blessed HPC approach but needs the user's code to cooperate, so it's at best a fast path, not the general mechanism. A hybrid is plausible.SIGKILL(the--signalflag and its grace period; also--requeue). Is the grace enough to dump potentially multi-GB memory images, and how do we wire signal handling SLURM → linkspan → checkpoint? Dumping state isn't free and the clock is the enemy: at parallel-FS write bandwidth on the order of GB/s, a few-GB image is seconds, but a large one can exceed a short grace period — so image size, write target, and the configured grace need to be reasoned about together, not separately.cuda-checkpoint+ CRIU is the thread to pull, but it's reasonable to scope GPU jobs out of v1 and detect-and-warn rather than silently fail.tmux/dtachso terminals survive and re-attach is likely cheaper and more robust than C/R-ing TTYs, and may carry most of the perceived seamlessness. Unsaved buffers likely come free from VS Code's hot-exit oncesessionIdis stable; the open question is whether a respawned extension host is acceptable.Bonus: Resume onto a different partition or resource configuration
Once same-config resume works, the same machinery enables migration: checkpoint on one resource and restore on another — e.g. a researcher develops on a small interactive node, then moves the session onto a high-performance/GPU partition without losing running processes, and back again.
This is strictly a second milestone — it adds heterogeneity constraints that don't exist in the same-config case and shouldn't block the base: CPU/microarchitecture differences can make a checkpoint non-restorable on the target; moving a process holding GPU state is the hard GPU-C/R problem again; glibc/kernel/library skew between partitions can break restore; and identity (IP, hostname, namespaces) all remaps, though the stable-
sessionIdtrick carries over. Build it only once the base is solid, and expect to constrain it to compatible partition pairs at first.Reference Material
In-repo: cs-bridge "How It Works" and CONTRIBUTING.md (session model,
sessions.json, per-session SSH config); linkspan README (workflow engine, REST API,internal/process/,subsystems/layout).External: DMTCP · CRIU · NVIDIA cuda-checkpoint · SLURM
--signal/grace/--requeue· runc/Podman/Apptainer ·tmux/dtach.Suggested start: checkpoint and restore a single CPU-bound process under DMTCP across a real SLURM walltime, driven by a linkspan workflow action and triggered by cs-bridge's existing
sacctpolling — enough to validate the spine before terminals, editor state, or migration. The two gating questions are unprivileged C/R and the grace window; everything else follows.