Multilinguality is crucial for extending recent advancements in language
modelling to diverse linguistic communities. To maintain high performance while
representing multiple languages, multilingual models ideally align
representations, allowing what is learned in one language to generalise to
others. Prior research has emphasised the importance of parallel data and
shared vocabulary elements as key factors for such alignment. In this study, we
investigate an unintuitive novel driver of cross-lingual generalisation:
language imbalance. In controlled experiments on perfectly equivalent cloned
languages, we observe that the existence of a predominant language during
training boosts the performance of less frequent languages and leads to
stronger alignment of model representations across languages. Furthermore, we
find that this trend is amplified with scale: with large enough models or long
enough training, we observe that bilingual training data with a 90/10 language
split yields better performance on both languages than a balanced 50/50 split.
Building on these insights, we design training schemes that can improve
performance in all cloned languages, even without altering the training data.
As we extend our analysis to real languages, we find that infrequent languages
still benefit from frequent ones, yet whether language imbalance causes
cross-lingual generalisation there is not conclusive.

多语种模型的研究指出，语言不平衡是跨语言泛化的一个新的驱动因素，在实验中观察到，在训练中存在一种主要语言可以提升较不频繁的语言的性能，同时加强模型在不同语言间的表示对齐，同时本研究还提出了一些训练方案来改善克隆语言的性能。

语言失衡可以促进跨语言泛化

Language Imbalance Can Boost Cross-lingual Generalisation

The vast majority of today's large language models are English-centric,
having been pretrained predominantly on English text. Yet, in order to meet
user expectations, models need to be able to respond appropriately in multiple
languages once deployed in downstream applications. Given limited exposure to
other languages during pretraining, cross-lingual transfer is important for
achieving decent performance in non-English settings. In this work, we
investigate just how much multilinguality is required during finetuning to
elicit strong cross-lingual generalisation across a range of tasks and target
languages. We find that, compared to English-only finetuning, multilingual
instruction tuning with as few as three languages significantly improves a
model's cross-lingual transfer abilities on generative tasks that assume
input/output language agreement, while being of less importance for highly
structured tasks. Our code and data is available at
this https URL

通过与只使用英文进行微调的模型相比较，研究发现只使用三种语言进行多语种微调可以显著提高模型在生成式任务上的跨语言转移能力，而在高度结构化的任务上影响较小。