Annually, research teams spend large amounts of money to evaluate the quality
of machine translation systems (WMT, inter alia). This is expensive because it
requires detailed human labor. The recently proposed annotation protocol, Error
Span Annotation (ESA), has annotators marking erroneous parts of the
translation. In our work, we help the annotators by pre-filling the span
annotations with automatic quality estimation. With AI assistance, we obtain
more detailed annotations while cutting down the time per span annotation by
half (71s/error span $\rightarrow$ 31s/error span). The biggest advantage of
ESA$^\mathrm{AI}$ protocol is an accurate priming of annotators (pre-filled
error spans) before they assign the final score as opposed to starting from
scratch. In addition, the annotation budget can be reduced by up to 24% with
filtering of examples that the AI deems to be very likely to be correct.

机器翻译系统的质量评估被提出的错误范围标注协议辅助，使用自动质量估计填充范围注释可以提供更详细的注释并将每个范围注释的时间缩短了一半，此外还可以减少最多 24% 的注释预算。

人工智能辅助机器翻译的人工评估

AI-Assisted Human Evaluation of Machine Translation

High-quality Machine Translation (MT) evaluation relies heavily on human
judgments. Comprehensive error classification methods, such as Multidimensional
Quality Metrics (MQM), are expensive as they are time-consuming and can only be
done by experts, whose availability may be limited especially for low-resource
languages. On the other hand, just assigning overall scores, like Direct
Assessment (DA), is simpler and faster and can be done by translators of any
level, but are less reliable. In this paper, we introduce Error Span Annotation
(ESA), a human evaluation protocol which combines the continuous rating of DA
with the high-level error severity span marking of MQM. We validate ESA by
comparing it to MQM and DA for 12 MT systems and one human reference
translation (English to German) from WMT23. The results show that ESA offers
faster and cheaper annotations than MQM at the same quality level, without the
requirement of expensive MQM experts.

机器翻译评估中，综合的错误分类方法如多维度质量度量 (MQM) 需要耗费大量时间和专家参与，而只赋予整体评分，如直接评估 (DA)，虽然简单快捷但可靠性较低。本文提出一种人工评估协议 —— 错误跨度标注 (ESA)，将 DA 的连续评分与 MQM 的高级错误严重程度标记相结合。通过与 12 个 MT 系统和一份人工参考翻译 (英语到德语) 从 WMT23 的 MQM 和 DA 进行比较，验证了 ESA 的有效性。结果显示 ESA 在具有相同质量水平的情况下比 MQM 更快且更便宜，无需昂贵的 MQM 专家。