DisagMoE: Computation-Communication overlapped MoE Training via Disaggregated AF-Pipe Parallelism

Authors: Zhichen Zeng, Chi-Chih Chang, Jiayi Wang et al.

arXiv · PDF

Summary

The authors design DisagMoE, a system for training mixture-of-experts (MoE) LLMs more efficiently by separating attention and expert (feed-forward) layers onto different GPU groups and pipelining their communication. Standard expert-parallelism training hits network bottlenecks because all-to-all communication (routing tokens to different experts across nodes) doesn't overlap well with computation. DisagMoE disaggregates the model so attention and FFN layers run on disjoint hardware, uses a multi-stage pipeline with unidirectional communication, and balances computation-to-communication ratios across the groups. On 16-node H800 clusters, DisagMoE delivers up to 1.8× training speedup.

Main takeaways:

• MoE models scale by activating only a subset of experts per token, but training them with expert parallelism causes severe all-to-all communication stalls.
• Prior overlap techniques can't fully hide communication because attention and FFN layers have different computation-to-communication ratios.
• DisagMoE splits attention and FFN onto separate GPU groups, uses a unidirectional many-to-many pipeline, and applies a roofline model to allocate bandwidth optimally.
• Implemented in Megatron-LM, DisagMoE achieves up to 1.8× speedup on multi-node clusters.
• The method is purely about distributed training infrastructure, not model behavior or alignment.

Relevance

Not directly related to my persona/midtraining work — this is a distributed-systems paper about MoE training efficiency, not about behavioral installation, fine-tuning effects, or persona geometry.

Threat model

Potential threat/caveat for clean result "Fine-tuning one persona on a two-marker chunk and another on the start marker plants the end marker at every donor answer's end, not chained to the start (LOW confidence)": this item discusses evaluation.

Abstract

arXiv:2605.11005v1 Announce Type: new Abstract: Mixture-of-experts (MoE) architectures enable trillion-parameter LLMs with sparsely activated experts. Expert parallelism (EP) is a widely adopted MoE training strategy, but it suffers from severe all-to-all communication bottlenecks, which is exaggerated by the limited inter-node network bandwidth as the growing model size requires distributing experts across GPU nodes. Prior work focused on overlapping these all-to-all communications with feed-forward network (FFN) and self-attention computations, which often leaves residual network-bound stalls due to inherent imbalance in attention and FFN layers' computation-communication ratios. We present DisagMoE, a disaggregated MoE training system that jointly optimizes model placement and scheduling for maximal efficiency. DisagMoE separates attention and FFN layers into disjoint GPU groups, introduces a multi-stage pipeline with uni-directional, many-to-many communications, and employs a computation-communication roofline model to balance GPU and network bandwidth allocation among the attention and FFN groups. DisagMoE is implemented on Megatron-LM, and evaluation shows that DisagMoE improves training efficiency across multiple MoE models with up to 1.8x speedup on 16-node 8xH800 clusters.