Although neural-based machine translation evaluation metrics, such as COMET
or BLEURT, have achieved strong correlations with human judgements, they are
sometimes unreliable in detecting certain phenomena that can be considered as
critical errors, such as deviations in entities and numbers. In contrast,
traditional evaluation metrics, such as BLEU or chrF, which measure lexical or
character overlap between translation hypotheses and human references, have
lower correlations with human judgements but are sensitive to such deviations.
In this paper, we investigate several ways of combining the two approaches in
order to increase robustness of state-of-the-art evaluation methods to
translations with critical errors. We show that by using additional information
during training, such as sentence-level features and word-level tags, the
trained metrics improve their capability to penalize translations with specific
troublesome phenomena, which leads to gains in correlation with human judgments
and on recent challenge sets on several language pairs.

本研究探讨将传统评估方法如 BLEU 或 chrF 与基于神经网络的机器翻译评估方法如 COMET 或 BLEURT 相结合，通过使用附加的诸如句子级特征和单词级标签等额外信息训练评估指标，提高了最新状态下的机器翻译评估方法的稳健性，从而在几种语言对上提高了与人类判断的相关性和在挑战数据集上获得了更好的表现。

BLEU Meets COMET: 结合词汇和神经度量以实现机器翻译评估的稳健性

BLEU Meets COMET: Combining Lexical and Neural Metrics Towards Robust  Machine Translation Evaluation

In this paper we present the first neural-based machine translation system
trained to translate between standard national varieties of the same language.
We take the pair Brazilian - European Portuguese as an example and compare the
performance of this method to a phrase-based statistical machine translation
system. We report a performance improvement of 0.9 BLEU points in translating
from European to Brazilian Portuguese and 0.2 BLEU points when translating in
the opposite direction. We also carried out a human evaluation experiment with
native speakers of Brazilian Portuguese which indicates that humans prefer the
output produced by the neural-based system in comparison to the statistical
system.

本文介绍了第一个基于神经网络的机器翻译系统，用于翻译同一语言的标准国别语言种。我们以巴西 - 欧洲葡萄牙语为例，将该方法的性能与短语统计机器翻译系统进行比较。结果显示，在从欧洲葡萄牙语翻译到巴西葡萄牙语时，BLEU 分数提高了 0.9 分，在相反方向翻译时提高了 0.2 分。我们还进行了一项面向巴西葡萄牙语母语者的人工评估实验，结果表明与统计系统相比，人们更喜欢神经系统生成的输出。