With the fast development of Machine Translation (MT) systems, especially the
new boost from Neural MT (NMT) models, the MT output quality has reached a new
level of accuracy. However, many researchers criticised that the current
popular evaluation metrics such as BLEU can not correctly distinguish the
state-of-the-art NMT systems regarding quality differences. In this short
paper, we describe the design and implementation of a linguistically motivated
human-in-the-loop evaluation metric looking into idiomatic and terminological
Multi-word Expressions (MWEs). MWEs have played a bottleneck in many Natural
Language Processing (NLP) tasks including MT. MWEs can be used as one of the
main factors to distinguish different MT systems by looking into their
capabilities in recognising and translating MWEs in an accurate and meaning
equivalent manner.

本文介绍了一种基于多词表达式的人机协作评估度量方法，旨在评估机器翻译系统中多词表达式的准确性和语义等价性，以此作为评估机器翻译系统的指标。

HilMeMe: 一种基于人机交互的机器翻译准确度度量方法，关注多词表达

HilMeMe: A Human-in-the-Loop Machine Translation Evaluation Metric Looking into Multi-Word Expressions

We present MLQE-PE, a new dataset for Machine Translation (MT) Quality
Estimation (QE) and Automatic Post-Editing (APE). The dataset contains eleven
language pairs, with human labels for up to 10,000 translations per language
pair in the following formats: sentence-level direct assessments and
post-editing effort, and word-level good/bad labels. It also contains the
post-edited sentences, as well as titles of the articles where the sentences
were extracted from, and the neural MT models used to translate the text.

MLQE-PE 为机器翻译质量评估和自动后编辑提供了新数据集，包含 11 种语言对，针对每种语言对的 10000 个翻译提供了人类标签，包括句子级别的直接评估和后编辑努力以及单词级别的好 / 坏标签，同时还提供了后编辑的句子、标题以及用于翻译文本的神经机器翻译模型。