In this paper, we elaborate upon recipes for building multilingual
representation models that are not only competitive with existing
state-of-the-art models but are also more parameter efficient, thereby
promoting better adoption in resource-constrained scenarios and practical
applications. We show that going beyond English-centric bitexts, coupled with a
novel sampling strategy aimed at reducing under-utilization of training data,
substantially boosts performance across model sizes for both Electra and MLM
pre-training objectives. We introduce XY-LENT: X-Y bitext enhanced Language
ENcodings using Transformers which not only achieves state-of-the-art
performance over 5 cross-lingual tasks within all model size bands, is also
competitive across bands. Our XY-LENT XL variant outperforms XLM-RXXL and
exhibits competitive performance with mT5 XXL while being 5x and 6x smaller
respectively. We then show that our proposed method helps ameliorate the curse
of multilinguality, with the XY-LENT XL achieving 99.3% GLUE performance and
98.5% SQuAD 2.0 performance compared to a SoTA English only model in the same
size band. We then analyze our models performance on extremely low resource
languages and posit that scaling alone may not be sufficient for improving the
performance in this scenario

本文介绍了一种构建多语言表示模型的方法，该方法不仅可以与现有的最先进的模型竞争，而且还更加参数高效，从而促进了资源受限情况和实际应用的更好采用。

更好的多语言表示学习：超越以英语为中心的双语语料

Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning

In cross-lingual text classification, one seeks to exploit labeled data from
one language to train a text classification model that can then be applied to a
completely different language. Recent multilingual representation models have
made it much easier to achieve this. Still, there may still be subtle
differences between languages that are neglected when doing so. To address
this, we present a semi-supervised adversarial training process that minimizes
the maximal loss for label-preserving input perturbations. The resulting model
then serves as a teacher to induce labels for unlabeled target language samples
that can be used during further adversarial training, allowing us to gradually
adapt our model to the target language. Compared with a number of strong
baselines, we observe significant gains in effectiveness on document and intent
classification for a diverse set of languages.

本研究提出了一种半监督敌对训练过程，通过最小化标签保留输入扰动的最大损失来改进跨语言文本分类，进而为目标语言样本诱导标签以进一步适应模型。与多个强基线比较，我们观察到了在各种语言的文档和意图分类上的显著增益。