Analysing translation quality in regards to specific linguistic phenomena has historically been difficult and time-consuming. Neural machine translation has the attractive property that it can produce scores for arbitrary translations, and we propose a novel method to assess how well NMT systems model specific linguistic phenomena such as agreement over long distances, the production of novel words, and the faithful translation of polarity. The core idea is that we measure whether a reference translation is more probable under a NMT model than a contrastive translation which introduces a specific type of error. We present LingEval90, a large-scale data set of 90000 contrastive translation pairs based on the WMT English->German translation task, with errors automatically created with simple rules. We report a number of baseline results, and find that recently introduced character-level NMT systems perform better at transliteration than models with BPE segmentation, but perform more poorly at morphosyntactic agreement, and translating discontiguous units of meaning.

通过对神经机器翻译实现对特定语言现象的质量评估，我们提出了一种新方法并给出了包含97000对用于WMT英语->德语翻译任务的对照翻译数据集LingEval97，对字符级别和字节对编码（BPE）分割模型的实验结果显示，前者在翻译转写方面表现更好，但在形态句法协议和翻译非连续的意义单元方面表现较差。

字符级神经机器翻译的语法性如何？通过对比翻译对评估机器翻译质量