Pretrained language models (PLMs) display impressive performances and have captured the attention of the NLP community. Establishing the best practices in pretraining has therefore become a major point of focus for much of NLP research -- especially since the insights developed for monolingual English models need not carry to more complex multilingual. One significant caveat of the current state of the art is that different works are rarely comparable: they often discuss different parameter counts, training data, and evaluation methodology. This paper proposes a comparison of multilingual pretraining objectives in a controlled methodological environment. We ensure that training data and model architectures are comparable, and discuss the downstream performances across 6 languages that we observe in probing and fine-tuning scenarios. We make two key observations: (1) the architecture dictates which pretraining objective is optimal; (2) multilingual translation is a very effective pre-training objective under the right conditions. We make our code, data, and model weights available at \texttt{\url{https://github.com/Helsinki-NLP/lm-vs-mt}}.

本研究解决了多语言预训练目标比较中存在的方法学不统一的问题。通过在控制环境下比较多种预训练目标，察觉到模型架构决定了最优目标，同时在特定条件下，多语言翻译作为预训练目标表现出有效性。这一发现对多语言模型的构建具有重要影响。

两个堆栈胜过一个：语言建模与翻译作为多语言预训练目标的比较