We study the effect of one type of imbalance often present in real-life
multilingual classification datasets: an uneven distribution of labels across
languages. We show evidence that fine-tuning a transformer-based Large Language
Model (LLM) on a dataset with this imbalance leads to worse performance, a more
pronounced separation of languages in the latent space, and the promotion of
uninformative features. We modify the traditional class weighing approach to
imbalance by calculating class weights separately for each language and show
that this helps mitigate those detrimental effects. These results create
awareness of the negative effects of language-specific class imbalance in
multilingual fine-tuning and the way in which the model learns to rely on the
separation of languages to perform the task.

我们研究了现实生活中多语种分类数据集中常见的一种不平衡现象：标签在不同语言之间的分布不均。我们通过证据表明，在这种不平衡的数据集上微调基于 Transformer 的 Large Language Model（LLM）会导致性能下降、隐空间中语言间的差异更加明显以及非信息性特征的促进。我们修改了传统的类别加权方法，通过为每种语言单独计算类别权重来缓解这些不利影响。这些结果引起人们对多语种微调中语言特定类别不平衡带来的负面影响以及模型在执行任务时依赖语言分离的认识。