Massively multilingual machine translation (MT) has shown impressive
capabilities, including zero and few-shot translation between low-resource
language pairs. However, these models are often evaluated on high-resource
languages with the assumption that they generalize to low-resource ones. The
difficulty of evaluating MT models on low-resource pairs is often due to lack
of standardized evaluation datasets. In this paper, we present MENYO-20k, the
first multi-domain parallel corpus with a special focus on clean orthography
for Yor\`ub\'a--English with standardized train-test splits for benchmarking.
We provide several neural MT benchmarks and compare them to the performance of
popular pre-trained (massively multilingual) MT models both for the
heterogeneous test set and its subdomains. Since these pre-trained models use
huge amounts of data with uncertain quality, we also analyze the effect of
diacritics, a major characteristic of Yor\`ub\'a, in the training data. We
investigate how and when this training condition affects the final quality and
intelligibility of a translation. Our models outperform massively multilingual
models such as Google ($+8.7$ BLEU) and Facebook M2M ($+9.1$ BLEU) when
translating to Yor\`ub\'a, setting a high quality benchmark for future
research.

本研究提出了 MENYO-20k，这是一个首个多领域平行语料库，专注于干净的约鲁巴语和英语，为基准测试提供标准化的训练集 - 测试集划分。研究人员提供了几个神经机器翻译基准测试，并将它们与流行的预训练（大规模多语种）机器翻译模型的性能进行了比较。出乎意料的是，他们的模型效果超过了谷歌（+8.7 BLEU）和 Facebook M2M（+9.1 BLEU）这样的大规模多语种模型，为未来的研究设定了高质量的基准。