Almost all frameworks for the manual or automatic evaluation of machine
translation characterize the quality of an MT output with a single number. An
exception is the Multidimensional Quality Metrics (MQM) framework which offers
a fine-grained ontology of quality dimensions for scoring (such as style,
fluency, accuracy, and terminology). Previous studies have demonstrated the
feasibility of MQM annotation but there are, to our knowledge, no computational
models that predict MQM scores for novel texts, due to a lack of resources. In
this paper, we address these shortcomings by (a) providing a 1200-sentence MQM
evaluation benchmark for the language pair English-Korean and (b) reframing MT
evaluation as the multi-task problem of simultaneously predicting several MQM
scores using SOTA language models, both in a reference-based MT evaluation
setup and a reference-free quality estimation (QE) setup. We find that
reference-free setup outperforms its counterpart in the style dimension while
reference-based models retain an edge regarding accuracy. Overall, RemBERT
emerges as the most promising model. Through our evaluation, we offer an
insight into the translation quality in a more fine-grained, interpretable
manner.

通过提供一个英韩语言对的 1200 句 MQM 评估基准，将机器翻译评估重新定义为使用 SOTA 语言模型同时预测多个 MQM 分数的多任务问题，在参考 MT 评估和无参考质量估计（QE）设置中，我们发现无参考设置在风格维度上优于参考设置，而参考模型在准确性方面保持优势，总体上，RemBERT 是最有希望的模型。通过我们的评估，以一种更精细化、可解释的方式提供了关于翻译质量的见解。