Skip to content

Fix optimizer checkpoint state round-trip - #15

Open
koriyoshi2041 wants to merge 2 commits into
THUDM:mainfrom
koriyoshi2041:fix-optimizer-checkpoint-state
Open

koriyoshi2041 wants to merge 2 commits into
THUDM:mainfrom
koriyoshi2041:fix-optimizer-checkpoint-state

Conversation

@koriyoshi2041

@koriyoshi2041 koriyoshi2041 commented Sep 1, 2026 •

Copy link
Copy Markdown

Fixes #9.

The optimizer wrapper now saves its own perturbation state, wrapper attributes, scheduler state, and the wrapped base optimizer in a versioned checkpoint. Loading keeps compatibility with legacy base-only checkpoints and rebinds the shared parameter groups.

The scheduler follow-up adds explicit phase round-tripping for INFTY schedulers and for the wrapped PyTorch scheduler used by ProportionScheduler. The resume regression advances both continuous and restored paths and compares the next scheduled value and LookSAM parameter update.

Validation:

  • uv run --frozen --with pytest --with pyyaml pytest -q — 33 passed
  • uv run --frozen --with pytest pytest tests/optim/test_geometry_reshaping.py -q — 4 passed
  • git diff --check

Known boundary: runtime closures are deliberately reattached with set_closure() rather than serialized. The separate LookSAM old_g / old_grad mismatch is not part of this checkpoint fix.

@koriyoshi2041

Copy link
Copy Markdown
Author

Small follow-up after rechecking the current head (ebf43ab): the PR remains cleanly mergeable. The focused optimizer suite passed 10/10 when I submitted it, including versioned and legacy checkpoint loading plus next-step numerical equivalence after restore. Happy to add narrower coverage if there is a checkpoint-compatibility case you want exercised before review.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Incomplete state_dict / load_state_dict coverage in InftyBaseOptimizer

1 participant