[Feature][Kernel]FusedMoE LoRA

[wcwuwc]

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.
  FIX [Feature]: Add LoRA support for gpt-oss model #23610

Purpose

This PR is an experimental extension to FusedMoE, aiming to support parallel inference with multiple LoRA models.

✅ Completed:

• Verified to work under both TP and torch.compile modes.
• Verified correctness on the DeepSeek-V2-Lite-Chat model.
• Compatible with PackedLoRALayerWeights to load LoRA weights of all experts.
• With test coverage.

❌ Work in progress:

• Kernel generalization: experts more than 256, support for quantization and compatibility with various GPU architectures.

Any suggestions are welcome！

Test Plan

vllm serve deepseek-ai/DeepSeek-V2-Lite-Chat --trust-remote-code -tp 2  --enable-lora --max-loras 6 --lora-modules  lora1=wuchen01/DeepSeek-V2-Lite-Chat-All-LoRA --max-lora-rank 64 

curl --location 'http://127.0.0.1:8000/v1/chat/completions' \
--header 'Content-Type: application/json' \
--data '{
    "stream":false,
    "messages": [
        {"role": "system", "content": "You are a helpful assistant."},
        {
            "role": "user",
            "content": "who are you?"
        }
    ],
    "model": "lora1"
    }'

Test Result

(Optional) Documentation Update

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @CNTRYROA.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[gemini-code-assist]

Code Review

This pull request introduces an experimental extension for FusedMoE to support parallel inference with multiple LoRA models. The changes are extensive, touching CUDA kernels, Triton kernels, and various Python layers for model execution and LoRA management. The core idea is to enable LoRA adapters for Mixture-of-Experts layers, which is a significant feature enhancement.

My review focuses on the correctness and robustness of the new kernels and their integration. I've identified a few critical and high-severity issues that should be addressed:

• A race condition in the new CUDA kernel (moe_lora_align_sum_kernels.cu) could lead to incorrect behavior or memory corruption.
• Incorrect shared memory calculation in the same CUDA kernel could lead to resource exhaustion and launch failures.
• Hardcoded values for the number of experts and LoRA rank in the Triton kernels and Python wrappers limit the general applicability of this feature.

These issues are important to fix to ensure the stability and correctness of this new feature. The PR is a work-in-progress, and addressing these points will significantly improve its quality and readiness for merging.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @CNTRYROA.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[jeejeelee]

LGTM
Thank you to everyone who contributed to this PR

[jeejeelee]

@Yikun @xuechendi @vanbasten23 this PR might affect LoRA in your backend

[xuechendi]

@jeejeelee , oh, Thanks for the info, will check from our backend