Popular benchmarks (e.g., XNLI) used to evaluate cross-lingual language
understanding consist of parallel versions of English evaluation sets in
multiple target languages created with the help of professional translators.
When creating such parallel data, it is critical to ensure high-quality
translations for all target languages for an accurate characterization of
cross-lingual transfer. In this work, we find that translation inconsistencies
do exist and interestingly they disproportionally impact low-resource languages
in XNLI. To identify such inconsistencies, we propose measuring the gap in
performance between zero-shot evaluations on the human-translated and
machine-translated target text across multiple target languages; relatively
large gaps are indicative of translation errors. We also corroborate that
translation errors exist for two target languages, namely Hindi and Urdu, by
doing a manual reannotation of human-translated test instances in these two
languages and finding poor agreement with the original English labels these
instances were supposed to inherit.

在评估跨语言语言理解的常用基准（如 XNLI）中，通过专业翻译人员创建用于多个目标语言的英文评估集的平行版本非常重要，以确保所有目标语言的高质量翻译，以准确地进行跨语言转移的表征。本研究发现，存在翻译的不一致性，并且这些不一致性在 XNLI 中对于低资源语言具有不成比例的影响。通过在多个目标语言的人工翻译和机器翻译目标文本之间进行零 - shot 评估的性能差距来识别这种不一致性，表现出相对较大的差距即为翻译错误的指示。此外，通过对印地语和乌尔都语这两种目标语言进行人工重新注释的方式，我们证实了翻译错误的存在，并发现这些实例与其原始的英文标签之间存在较差的一致性。

翻译错误对跨语言学习中的低资源语言有重大影响

Translation Errors Significantly Impact Low-Resource Languages in  Cross-Lingual Learning

State-of-the-art natural language processing systems rely on supervision in
the form of annotated data to learn competent models. These models are
generally trained on data in a single language (usually English), and cannot be
directly used beyond that language. Since collecting data in every language is
not realistic, there has been a growing interest in cross-lingual language
understanding (XLU) and low-resource cross-language transfer. In this work, we
construct an evaluation set for XLU by extending the development and test sets
of the Multi-Genre Natural Language Inference Corpus (MultiNLI) to 15
languages, including low-resource languages such as Swahili and Urdu. We hope
that our dataset, dubbed XNLI, will catalyze research in cross-lingual sentence
understanding by providing an informative standard evaluation task. In
addition, we provide several baselines for multilingual sentence understanding,
including two based on machine translation systems, and two that use parallel
data to train aligned multilingual bag-of-words and LSTM encoders. We find that
XNLI represents a practical and challenging evaluation suite, and that directly
translating the test data yields the best performance among available
baselines.

本文提出了 XNLI 数据集用于跨语言句子理解的评估，并研究了多语言句子理解的基准和挑战。