Sparsely gated Mixture of Experts (MoE) models have been shown to be a
compute-efficient method to scale model capacity for multilingual machine
translation. However, for low-resource tasks, MoE models severely over-fit. We
show effective regularization strategies, namely dropout techniques for MoE
layers in EOM and FOM, Conditional MoE Routing and Curriculum Learning methods
that prevent over-fitting and improve the performance of MoE models on
low-resource tasks without adversely affecting high-resource tasks. On a
massively multilingual machine translation benchmark, our strategies result in
about +1 chrF++ improvement in very low resource language pairs. We perform an
extensive analysis of the learned MoE routing to better understand the impact
of our regularization methods and how we can improve them.

通过引入 dropout 和条件化的 MoE 路由和课程学习技术，本研究针对低资源任务下 MoE 模型的严重过拟合问题，提出有效的正则化策略，成功提高了 MoE 模型在低资源任务中的性能，并在大规模多语种机器翻译基准测试中取得了巨大的改进。

多语言机器翻译中修复低资源语言的 MoE 过拟合问题

Fixing MoE Over-Fitting on Low-Resource Languages in Multilingual Machine Translation

Driven by the goal of eradicating language barriers on a global scale,
machine translation has solidified itself as a key focus of artificial
intelligence research today. However, such efforts have coalesced around a
small subset of languages, leaving behind the vast majority of mostly
low-resource languages. What does it take to break the 200 language barrier
while ensuring safe, high quality results, all while keeping ethical
considerations in mind? In No Language Left Behind, we took on this challenge
by first contextualizing the need for low-resource language translation support
through exploratory interviews with native speakers. Then, we created datasets
and models aimed at narrowing the performance gap between low and high-resource
languages. More specifically, we developed a conditional compute model based on
Sparsely Gated Mixture of Experts that is trained on data obtained with novel
and effective data mining techniques tailored for low-resource languages. We
propose multiple architectural and training improvements to counteract
overfitting while training on thousands of tasks. Critically, we evaluated the
performance of over 40,000 different translation directions using a
human-translated benchmark, Flores-200, and combined human evaluation with a
novel toxicity benchmark covering all languages in Flores-200 to assess
translation safety. Our model achieves an improvement of 44% BLEU relative to
the previous state-of-the-art, laying important groundwork towards realizing a
universal translation system. Finally, we open source all contributions
described in this work, accessible at
this https URL.

本文介绍了一项针对低资源语言的研究，使用 Sparsely Gated Mixture of Experts 模型结合新的数据挖掘技术进行训练，从而实现了机器翻译中对于低资源语言的支持并提高了 BLEU 值。