Conversation
Add dumper CLI arguments (--dumper-enable, --dumper-dir, per-phase config), dumper_utils.py for SGLang/Megatron dumper integration, model.py hooks for forward-only and forward-backward phases, rollout env var plumbing, source patcher wiring in training actors, and basic e2e test.
fzyzcjy
requested review from
guapisolo,
maocheng23,
yueming-yuan and
yushengsu-thu
as code owners
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly enhances the debugging and analysis capabilities within the Miles framework by integrating a powerful data dumper. The primary goal is to provide developers with detailed insights into the internal states of models and data flows during both inference and training phases. This integration allows for granular control over what data is dumped and when, facilitating easier identification and resolution of issues, and supporting in-depth post-hoc analysis without altering the core logic of the system. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a new dumper utility into the miles framework, supporting FSDP and Megatron training backends, as well as SGLang inference. The changes include new command-line arguments, a dumper_utils.py file, and comprehensive tests. The implementation handles distributed environments and provides flexible configuration. A limitation regarding virtual pipeline parallelism in the _extract_model method is noted, with a suggestion to add a TODO for future improvement.
| def _extract_model(model: Sequence[torch.nn.Module]) -> torch.nn.Module: | ||
| assert ( | ||
| len(model) == 1 | ||
| ), f"Dumper does not yet support virtual pipeline parallelism (got {len(model)} model chunks)" | ||
| return model[0] |
There was a problem hiding this comment.
The _extract_model method explicitly states that the dumper does not yet support virtual pipeline parallelism. Consider adding a TODO to address this limitation in future iterations, especially if virtual pipeline parallelism is a planned feature, to ensure full compatibility.
1 participant
No description provided.