Current Machine Translation (MT) systems achieve very good results on a growing variety of language pairs and datasets. However, they are known to produce fluent translation outputs that can contain important meaning errors, thus undermining their reliability in practice. Quality Estimation (QE) is the task of automatically assessing the performance of MT systems at test time. Thus, in order to be useful, QE systems should be able to detect such errors. However, this ability is yet to be tested in the current evaluation practices, where QE systems are assessed only in terms of their correlation with human judgements. In this work, we bridge this gap by proposing a general methodology for adversarial testing of QE for MT. First, we show that despite a high correlation with human judgements achieved by the recent SOTA, certain types of meaning errors are still problematic for QE to detect. Second, we show that on average, the ability of a given model to discriminate between meaning-preserving and meaning-altering perturbations is predictive of its overall performance, thus potentially allowing for comparing QE systems without relying on manual quality annotation.

该论文提出了一种对机器翻译中的质量评估进行敌对测试的方法，通过研究近期最优设备的评价体系，发现某些含有意义错误的翻译结果是难以被评估系统检测的。同时，该论文还研究了翻译结果保留原本含义和改变原本含义两种扰动的区别，并探讨了这种方法对于评估系统的多个领域可能产生的影响以及评估结果可行性的可比性。

推动正确按钮：对质量评估的对抗性评估