Recent advances have made non-autoregressive (NAT) translation comparable to
autoregressive methods (AT). However, their evaluation using BLEU has been
shown to weakly correlate with human annotations. Limited research compares
non-autoregressive translation and autoregressive translation comprehensively,
leaving uncertainty about the true proximity of NAT to AT. To address this gap,
we systematically evaluate four representative NAT methods across various
dimensions, including human evaluation. Our empirical results demonstrate that
despite narrowing the performance gap, state-of-the-art NAT still underperforms
AT under more reliable evaluation metrics. Furthermore, we discover that
explicitly modeling dependencies is crucial for generating natural language and
generalizing to out-of-distribution sequences.

最近的研究发现非自回归翻译方法与自回归方法在性能上存在一定差距，且传统的评估指标不太相关。本文通过系统评估四种非自回归方法，并结合人工评估，发现尽管性能差距在缩小，但当前领先的非自回归方法仍然不如自回归方法，在更可靠的评估指标下表现不佳。此外，我们还发现明确建模依赖关系对于生成自然语言并推广到其他序列是至关重要的。