[phyai] MagiAttention Backend

[chenghuaWang]

Summary

Add backend support for SandAI-org/MagiAttention, a distributed/context-parallel attention implementation designed for ultra-long-context training and heterogeneous attention masks.

Motivation

MagiAttention provides a scalable distributed attention backend with support for flexible mask patterns, computation load balancing, zero-redundant communication, and computation/communication overlap. Supporting it in phyai would make it easier to run long-context training workloads that require context parallelism and efficient distributed attention.

References

• https://github.com/SandAI-org/MagiAttention

[spidey-zyc]

May I help with it? I am an undergraduate, wish to practice my coding abilities
另外作者是不是中国人。