While machine translation evaluation has been studied primarily for
high-resource languages, there has been a recent interest in evaluation for
low-resource languages due to the increasing availability of data and models.
In this paper, we focus on a zero-shot evaluation setting focusing on
low-resource Indian languages, namely Assamese, Kannada, Maithili, and Punjabi.
We collect sufficient Multi-Dimensional Quality Metrics (MQM) and Direct
Assessment (DA) annotations to create test sets and meta-evaluate a plethora of
automatic evaluation metrics. We observe that even for learned metrics, which
are known to exhibit zero-shot performance, the Kendall Tau and Pearson
correlations with human annotations are only as high as 0.32 and 0.45.
Synthetic data approaches show mixed results and overall do not help close the
gap by much for these languages. This indicates that there is still a long way
to go for low-resource evaluation.

机器翻译评估在高资源语言上的研究已有所涉及，但基于近期数据和模型的增加，对低资源语言的评估也引起了兴趣。本文针对低资源的印度语言，即阿萨姆语、卡纳达语、迈蒂利语和旁遮普语，通过收集足够的多维度质量度量和直接评估注解，构建测试集并使用元评估方法对大量的自动评估度量进行评估。研究发现，即使对于已知具备零样本性能的学习度量，其与人类注解的肯德尔 τ 和皮尔逊相关系数仅有 0.32 和 0.45。合成数据方法显示了各种结果，总体上并没有在这些语言中有太大的帮助。这表明在低资源评估方面还有很长的路要走。