Recent studies have highlighted the potential of exploiting parallel corpora
to enhance multilingual large language models, improving performance in both
bilingual tasks, e.g., machine translation, and general-purpose tasks, e.g.,
text classification. Building upon these findings, our comprehensive study aims
to identify the most effective strategies for leveraging parallel corpora. We
investigate the impact of parallel corpora quality and quantity, training
objectives, and model size on the performance of multilingual large language
models enhanced with parallel corpora across diverse languages and tasks. Our
analysis reveals several key insights: (i) filtering noisy translations is
essential for effectively exploiting parallel corpora, while language
identification and short sentence filtering have little effect; (ii) even a
corpus containing just 10K parallel sentences can yield results comparable to
those obtained from much larger datasets; (iii) employing only the machine
translation objective yields the best results among various training objectives
and their combinations; (iv) larger multilingual language models benefit more
from parallel corpora than smaller models due to their stronger capacity for
cross-task transfer. Our study offers valuable insights into the optimal
utilization of parallel corpora to enhance multilingual large language models,
extending the generalizability of previous findings from limited languages and
tasks to a broader range of scenarios.

通过对平行语料库进行优化，可以提高多语言大型语言模型的性能，研究结果表明，过滤嘈杂的翻译是有效利用平行语料库的关键。

面向多语言大语言模型的平行语料利用方法

A Recipe of Parallel Corpora Exploitation for Multilingual Large  Language Models

With the success of large-scale pre-training and multilingual modeling in
Natural Language Processing (NLP), recent years have seen a proliferation of
large, web-mined text datasets covering hundreds of languages. We manually
audit the quality of 205 language-specific corpora released with five major
public datasets (CCAligned, ParaCrawl, WikiMatrix, OSCAR, mC4). Lower-resource
corpora have systematic issues: At least 15 corpora have no usable text, and a
significant fraction contains less than 50% sentences of acceptable quality. In
addition, many are mislabeled or use nonstandard/ambiguous language codes. We
demonstrate that these issues are easy to detect even for non-proficient
speakers, and supplement the human audit with automatic analyses. Finally, we
recommend techniques to evaluate and improve multilingual corpora and discuss
potential risks that come with low-quality data releases.

本文主要旨在探究如何评估及提高多语料库的质量，并针对少资源语言建议相关技术，同时也讨论低质量数据发布所可能带来的潜在风险。