Despite the increasing number of large and comprehensive machine translation
(MT) systems, evaluation of these methods in various languages has been
restrained by the lack of high-quality parallel corpora as well as engagement
with the people that speak these languages. In this study, we present an
evaluation of state-of-the-art approaches to training and evaluating MT systems
in 22 languages from the Turkic language family, most of which being extremely
under-explored. First, we adopt the TIL Corpus with a few key improvements to
the training and the evaluation sets. Then, we train 26 bilingual baselines as
well as a multi-way neural MT (MNMT) model using the corpus and perform an
extensive analysis using automatic metrics as well as human evaluations. We
find that the MNMT model outperforms almost all bilingual baselines in the
out-of-domain test sets and finetuning the model on a downstream task of a
single pair also results in a huge performance boost in both low- and
high-resource scenarios. Our attentive analysis of evaluation criteria for MT
models in Turkic languages also points to the necessity for further research in
this direction. We release the corpus splits, test sets as well as models to
the public.

本研究通过对土耳其语系 22 种语言的大规模机器翻译系统的培训和评估，发现 MNMT 模型在领域外测试集中的表现优于几乎所有双语基线，并在单对下游任务的微调中也获得了巨大的性能提升。