Automatic evaluation comparing candidate translations to human-generated
paraphrases of reference translations has recently been proposed by Freitag et
al. When used in place of original references, the paraphrased versions produce
metric scores that correlate better with human judgment. This effect holds for
a variety of different automatic metrics, and tends to favor natural
formulations over more literal (translationese) ones. In this paper we compare
the results of performing end-to-end system development using standard and
paraphrased references. With state-of-the-art English-German NMT components, we
show that tuning to paraphrased references produces a system that is
significantly better according to human judgment, but 5 BLEU points worse when
tested on standard references. Our work confirms the finding that paraphrased
references yield metric scores that correlate better with human judgment, and
demonstrates for the first time that using these scores for system development
can lead to significant improvements.

本文研究采用人工生成的同义词组进行翻译评估，结果表明使用同义词组可以更好地反映人类判断，同时使用同义词组进行系统开发可以产生显著的提高。