Sparse mixture of experts provides larger model capacity while requiring a
constant computational overhead. It employs the routing mechanism to distribute
input tokens to the best-matched experts according to their hidden
representations. However, learning such a routing mechanism encourages token
clustering around expert centroids, implying a trend toward representation
collapse. In this work, we propose to estimate the routing scores between
tokens and experts on a low-dimensional hypersphere. We conduct extensive
experiments on cross-lingual language model pre-training and fine-tuning on
downstream tasks. Experimental results across seven multilingual benchmarks
show that our method achieves consistent gains. We also present a comprehensive
analysis on the representation and routing behaviors of our models. Our method
alleviates the representation collapse issue and achieves more consistent
routing than the baseline mixture-of-experts methods.

该研究提出了一种在低维超球面上估计令牌和专家之间路由得分的方法，克服了表示塌陷问题并取得了更为一致路由的实验结果。

稀疏专家混合模型的表示崩溃

On the Representation Collapse of Sparse Mixture of Experts

In this work, we present an information-theoretic framework that formulates
cross-lingual language model pre-training as maximizing mutual information
between multilingual-multi-granularity texts. The unified view helps us to
better understand the existing methods for learning cross-lingual
representations. More importantly, inspired by the framework, we propose a new
pre-training task based on contrastive learning. Specifically, we regard a
bilingual sentence pair as two views of the same meaning and encourage their
encoded representations to be more similar than the negative examples. By
leveraging both monolingual and parallel corpora, we jointly train the pretext
tasks to improve the cross-lingual transferability of pre-trained models.
Experimental results on several benchmarks show that our approach achieves
considerably better performance. The code and pre-trained models are available
at this https URL

该研究提出了一种信息论框架，将跨语言语言模型预训练作为最大化多语言 - 多粒度文本之间的相互信息来表述，以提高预训练模型的跨语言可迁移性，并提出了一种基于对比学习的预训练任务，实现了更好的预训练模型性能。