Non-Autoregressive machine Translation (NAT) models have demonstrated
significant inference speedup but suffer from inferior translation accuracy.
The common practice to tackle the problem is transferring the Autoregressive
machine Translation (AT) knowledge to NAT models, e.g., with knowledge
distillation. In this work, we hypothesize and empirically verify that AT and
NAT encoders capture different linguistic properties of source sentences.
Therefore, we propose to adopt Multi-Task learning to transfer the AT knowledge
to NAT models through encoder sharing. Specifically, we take the AT model as an
auxiliary task to enhance NAT model performance. Experimental results on WMT14
English-German and WMT16 English-Romanian datasets show that the proposed
Multi-Task NAT achieves significant improvements over the baseline NAT models.
Furthermore, the performance on large-scale WMT19 and WMT20 English-German
datasets confirm the consistency of our proposed method. In addition,
experimental results demonstrate that our Multi-Task NAT is complementary to
knowledge distillation, the standard knowledge transfer method for NAT.

通过多任务学习和编码器共享，将自回归机器翻译的知识转移到非自回归机器翻译模型中，实现翻译质量的提升。

非自回归机器翻译共享编码器的多任务学习

Multi-Task Learning with Shared Encoder for Non-Autoregressive Machine  Translation

Much recent effort has been invested in non-autoregressive neural machine
translation, which appears to be an efficient alternative to state-of-the-art
autoregressive machine translation on modern GPUs. In contrast to the latter,
where generation is sequential, the former allows generation to be parallelized
across target token positions. Some of the latest non-autoregressive models
have achieved impressive translation quality-speed tradeoffs compared to
autoregressive baselines. In this work, we reexamine this tradeoff and argue
that autoregressive baselines can be substantially sped up without loss in
accuracy. Specifically, we study autoregressive models with encoders and
decoders of varied depths. Our extensive experiments show that given a
sufficiently deep encoder, a single-layer autoregressive decoder can
substantially outperform strong non-autoregressive models with comparable
inference speed. We show that the speed disadvantage for autoregressive
baselines compared to non-autoregressive methods has been overestimated in
three aspects: suboptimal layer allocation, insufficient speed measurement, and
lack of knowledge distillation. Our results establish a new protocol for future
research toward fast, accurate machine translation. Our code is available at
this https URL

本研究发现，在编码器足够深的情况下，单层自回归解码器可以显著地优于拥有相似推理速度的强大非自回归模型，并认为自回归基准与非自回归方法之间的速度劣势在某些方面被高估，这些结果为未来研究快速且准确的机器翻译建立了新的协议。