Despite the recent success of automatic metrics for assessing translation
quality, their application in evaluating the quality of machine-translated
chats has been limited. Unlike more structured texts like news, chat
conversations are often unstructured, short, and heavily reliant on contextual
information. This poses questions about the reliability of existing
sentence-level metrics in this domain as well as the role of context in
assessing the translation quality. Motivated by this, we conduct a
meta-evaluation of existing sentence-level automatic metrics, primarily
designed for structured domains such as news, to assess the quality of
machine-translated chats. We find that reference-free metrics lag behind
reference-based ones, especially when evaluating translation quality in
out-of-English settings. We then investigate how incorporating conversational
contextual information in these metrics affects their performance. Our findings
show that augmenting neural learned metrics with contextual information helps
improve correlation with human judgments in the reference-free scenario and
when evaluating translations in out-of-English settings. Finally, we propose a
new evaluation metric, Context-MQM, that utilizes bilingual context with a
large language model (LLM) and further validate that adding context helps even
for LLM-based evaluation metrics.

通过对自动度量、机器翻译聊天等领域进行元评估，我们发现引用自由度量相较于引用依赖度量滞后，尤其在评估英文以外的翻译质量时。我们研究了如何将对话上下文信息融入度量中，并发现将上下文信息与神经学习度量相结合有助于提高自由度量在无参考情景下与人类判断的相关性，以及在评估英文以外的翻译时的性能。最后，我们提出了一种新的评估度量方法 Context-MQM，并验证了添加上下文即使对基于大型语言模型（LLM）的评估度量也是有帮助的。