weight cache deamon - #1083
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 13d28b35cb
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| weight_cache_daemon_procs = None | ||
| if getattr(server_args, "weight_cache_mode", "off") == "daemon": | ||
| weight_cache_daemon_procs = _launch_weight_cache_daemons(server_args) |
There was a problem hiding this comment.
In engine-managed daemon mode, the Popen handles returned here are kept only in this local and are never retained or polled after _launch_subprocesses returns. If a weight-cache daemon crashes after schedulers have imported its CUDA IPC tensors, it can remain as an unreaped zombie under this parent; the IPC loader watchdog checks liveness with os.kill(pid, 0), which still succeeds for zombies, so the engine can continue using dangling IPC mappings instead of terminating. Keep these processes owned by the engine lifetime and reap/supervise them when they exit.
Useful? React with 👍 / 👎.
| weight_cache_mode = getattr(server_args, "weight_cache_mode", "off") | ||
| use_weight_cache = weight_cache_mode != "off" | ||
|
|
||
| # Create load config | ||
| load_config = LoadConfig( | ||
| load_format=server_args.load_format, | ||
| load_format=("ipc_cache" if use_weight_cache else server_args.load_format), |
There was a problem hiding this comment.
When speculative decoding uses a separate draft model, WeightLoader.load_model is called for the draft runner as well, but this global weight_cache_mode switch forces the draft load to use ipc_cache and the same per-rank socket. The daemon launcher only caches server_args.model, so the draft runner connects to the target model's daemon and fails the CacheConfig check instead of loading the draft weights from disk. Gate IPC loading to the target model or pass draft-aware cache settings.
Useful? React with 👍 / 👎.
1 participant
Summary
Test Plan