[Feat] Add support for MLP tensor splitting in the DeepSeek DenseFFN layer

[yuweivvv]

What this PR does / why we need it?

Motivation

In the DeepSeek-R1 architecture, the first 3 blocks utilize Dense Feed-Forward Network (DenseFFN) layers, while the subsequent 58 blocks are MoE layers. The Gate_Up_Proj and Down_Proj matrices in these DenseFFN layers have large dimensions ([7168, 36864] and [18432, 7168] respectively).

This results in significant memory overhead:
FP16/BF16: ~1.48 GB per layer × 3 layers = 4.44 GB
W8A8 Quantization: ~1.11 GB total

During the decoding phase, using a pure Data Parallel (DP) strategy requires each GPU to store the complete model weights for these non-MoE layers. This redundancy causes excessive VRAM usage on each device.

Proposed Changes

This PR implements partial Tensor Parallelism (TP) splitting for the memory-intensive DenseFFN weights. Instead of replicating the full weights across all devices (DP), we split these specific tensors across the available GPUs.

Impact Analysis

Memory Efficiency: significantly reduces the memory footprint of weights on individual cards.
Performance: The performance gain from parallelizing the large matrix multiplications outweighs the introduced communication overhead.
Conclusion: We achieve lower VRAM usage without compromising TPOT (Time Per Output Token).

Does this PR introduce any user-facing change?

This PR introduces a new parameter in additional_config.

--additional_config={"denseffn_tensor_parallel_size": 8}

How was this patch tested?

• vLLM version: v0.11.0
• vLLM main: vllm-project/vllm@2918c1b

[github-actions]

👋 Hi! Thank you for contributing to the vLLM Ascend project. The following points will speed up your PR merge:‌‌

• A PR should do only one thing, smaller PRs enable faster reviews.
• Every PR should include unit tests and end-to-end tests ‌to ensure it works and is not broken by other future PRs.
• Write the commit message by fulfilling the PR description to help reviewer and future developers understand.

If CI fails, you can run linting and testing checks locally according Contributing and Testing.

[gemini-code-assist]

Code Review

This pull request introduces tensor parallelism for the DenseFFN layers in DeepSeek models to reduce memory overhead on decode nodes. The approach of creating a dedicated tensor parallel group for these specific layers is sound. However, I've identified a few critical issues in the implementation. The configuration validation messages are contradictory, which could lead to user confusion. More importantly, the logic for selecting the new parallel group is too broad and would incorrectly apply it to MoE layers as well, which needs to be corrected. Please see the detailed comments for suggestions on how to fix these issues.