Widely used learned metrics for machine translation evaluation, such as COMET
and BLEURT, estimate the quality of a translation hypothesis by providing a
single sentence-level score. As such, they offer little insight into
translation errors (e.g., what are the errors and what is their severity). On
the other hand, generative large language models (LLMs) are amplifying the
adoption of more granular strategies to evaluation, attempting to detail and
categorize translation errors. In this work, we introduce xCOMET, an
open-source learned metric designed to bridge the gap between these approaches.
xCOMET integrates both sentence-level evaluation and error span detection
capabilities, exhibiting state-of-the-art performance across all types of
evaluation (sentence-level, system-level, and error span detection). Moreover,
it does so while highlighting and categorizing error spans, thus enriching the
quality assessment. We also provide a robustness analysis with stress tests,
and show that xCOMET is largely capable of identifying localized critical
errors and hallucinations.

本研究介绍了一种名为 xCOMET 的开源学习度量方法，它融合了语句级评估和错误跨度检测能力，并展现出在各种评估类型（语句级、系统级和错误跨度检测）方面的最先进性能，同时突出和分类错误跨度，从而丰富了质量评估。通过稳健性分析和压力测试，我们还表明 xCOMET 在识别局部关键错误和产生幻觉方面有很强的能力。