Analysing translation quality in regards to specific linguistic phenomena has
historically been difficult and time-consuming. Neural machine translation has
the attractive property that it can produce scores for arbitrary translations,
and we propose a novel method to assess how well NMT systems model specific
linguistic phenomena such as agreement over long distances, the production of
novel words, and the faithful translation of polarity. The core idea is that we
measure whether a reference translation is more probable under a NMT model than
a contrastive translation which introduces a specific type of error. We present
LingEval97, a large-scale data set of 97000 contrastive translation pairs based
on the WMT English->German translation task, with errors automatically created
with simple rules. We report results for a number of systems, and find that
recently introduced character-level NMT systems perform better at
transliteration than models with byte-pair encoding (BPE) segmentation, but
perform more poorly at morphosyntactic agreement, and translating discontiguous
units of meaning.

通过对神经机器翻译实现对特定语言现象的质量评估，我们提出了一种新方法并给出了包含 97000 对用于 WMT 英语 -> 德语翻译任务的对照翻译数据集 LingEval97，对字符级别和字节对编码（BPE）分割模型的实验结果显示，前者在翻译转写方面表现更好，但在形态句法协议和翻译非连续的意义单元方面表现较差。