Reset opponent environments when swapping historical policies

[bagel786]

Summary

When a historical opponent checkpoint is swapped in, reset every CPU environment assigned to that policy before the next rollout. Clear environment boundaries, recurrent state, reward state, and terminal state, then upload the refreshed buffers so episodes cannot span two different opponent policies.

Fixes #620.

Testing

• git diff --check passed.
• CUDA build/runtime validation was not available on this machine and should run in CI or on a CUDA host before merge.

---

Continues #672 — the fork repo backing that PR was accidentally deleted from this account, and GitHub blocks reopening a PR whose submitting repository was deleted (a restore request is being filed with GitHub Support). This PR resumes the identical work from the same commit (74ec736db70ace46181864774ea14601fa568eec); review discussion continues on the original PR.

[Copilot]

🔵 Needs a closer look

Environment matching must account for physical policy slots in multi-policy environments.

Pull request overview

Resets CPU environments and recurrent state when historical opponent policies are swapped.

Changes:

• Resets environment boundaries, rewards, terminals, and recurrent state.
• Uploads refreshed buffers before the next rollout.
• Applies resets after historical checkpoint swaps.

File summaries

File	Summary
src/pufferl.cu	Implements policy-specific environment resets; matching can fail for multi-policy environments because env->tag reflects only the maximum policy ID.

Review details

Suppressed comments (1)

src/pufferl.cu:1241

• env->tag is assigned as the maximum effective policy ID for an environment in env_setup, not as an identity for every policy used by its agents. Thus an environment containing policies [0, 1, 2] has tag 2 and is skipped when policy 1 is swapped, leaving that agent's recurrent state and environment trajectory on the old opponent. Determine whether any agent's physical slot maps to policy_idx via policy_layout, and reset the whole environment when it does.

        if (env->tag != policy_idx) {
            continue;

• Files reviewed: 1/1 changed files
• Comments generated: 0
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.