MoE facilitates the development of large models by making the computational
complexity of the model no longer scale linearly with increasing parameters.
The learning sparse gating network selects a set of experts for each token to
be processed; however, this may lead to differences in the number of tokens
processed by each expert over several successive iterations, i.e., the expert
load fluctuations, which reduces computational parallelization and resource
utilization. To this end, we traced and analyzed loads of each expert in the
training iterations for several large language models in this work, and defined
the transient state with "obvious load fluctuation" and the stable state with
"temporal locality". Moreover, given the characteristics of these two states
and the computational overhead, we deployed three classical prediction
algorithms that achieve accurate expert load prediction results. For the GPT3
350M model, the average error rates for predicting the expert load proportion
over the next 1,000 and 2,000 steps are approximately 1.3% and 1.8%,
respectively. This work can provide valuable guidance for expert placement or
resource allocation for MoE model training. Based on this work, we will propose
an expert placement scheme for transient and stable states in our coming work.

通过追踪和分析大型语言模型训练过程中每个专家的负载，对其负载波动进行了分类并定义了 “明显负载波动” 的瞬态状态和 “时间局部性” 的稳态状态，并应用经典的预测算法实现了准确的专家负载预测，从而为 MoE 模型训练提供了宝贵的专家安置或资源分配指导。