Efficient machine translation models are commercially important as they can
increase inference speeds, and reduce costs and carbon emissions. Recently,
there has been much interest in non-autoregressive (NAR) models, which promise
faster translation. In parallel to the research on NAR models, there have been
successful attempts to create optimized autoregressive models as part of the
WMT shared task on efficient translation. In this paper, we point out flaws in
the evaluation methodology present in the literature on NAR models and we
provide a fair comparison between a state-of-the-art NAR model and the
autoregressive submissions to the shared task. We make the case for consistent
evaluation of NAR models, and also for the importance of comparing NAR models
with other widely used methods for improving efficiency. We run experiments
with a connectionist-temporal-classification-based (CTC) NAR model implemented
in C++ and compare it with AR models using wall clock times. Our results show
that, although NAR models are faster on GPUs, with small batch sizes, they are
almost always slower under more realistic usage conditions. We call for more
realistic and extensive evaluation of NAR models in future work.

本研究对非自回归模型及自回归模型在机器翻译中的效率进行了探究，并提出了对非自回归模型的进一步实验和评测。