Traditionally, Machine Translation (MT) Evaluation has been treated as a
regression problem -- producing an absolute translation-quality score. This
approach has two limitations: i) the scores lack interpretability, and human
annotators struggle with giving consistent scores; ii) most scoring methods are
based on (reference, translation) pairs, limiting their applicability in
real-world scenarios where references are absent. In practice, we often care
about whether a new MT system is better or worse than some competitors. In
addition, reference-free MT evaluation is increasingly practical and necessary.
Unfortunately, these two practical considerations have yet to be jointly
explored. In this work, we formulate the reference-free MT evaluation into a
pairwise ranking problem. Given the source sentence and a pair of translations,
our system predicts which translation is better. In addition to proposing this
new formulation, we further show that this new paradigm can demonstrate
superior correlation with human judgments by merely using indirect supervision
from natural language inference and weak supervision from our synthetic data.
In the context of reference-free evaluation, MT-Ranker, trained without any
human annotations, achieves state-of-the-art results on the WMT Shared Metrics
Task benchmarks DARR20, MQM20, and MQM21. On a more challenging benchmark,
ACES, which contains fine-grained evaluation criteria such as addition,
omission, and mistranslation errors, MT-Ranker marks state-of-the-art against
reference-free as well as reference-based baselines.

机器翻译评估已被传统上视为回归问题，但这种方法存在两个限制：缺乏可解释性且人工评注者难以给出一致的分数；在真实的场景中，大多数评分方法基于（参考翻译）对，限制了它们的适用性。本研究提出一种新的评估方法，将无参考机器翻译评估转化为配对排序问题，并通过使用自然语言推理的间接监督和合成数据的弱监督来展示其与人类判断的卓越相关性。MT-Ranker 在无人工注释的情况下，在 DARR20、MQM20、MQM21 等 WMT Shared Metrics Task 基准测试上取得了最先进的结果，在包含细粒度评估标准的更有挑战的基准测试 ACES 上，MT-Ranker 在无参考和参考基准线上表现出最先进的水平。