Non-autoregressive approaches aim to improve the inference speed of
translation models, particularly those that generate output in a one-pass
forward manner. However, these approaches often suffer from a significant drop
in translation quality compared to autoregressive models. This paper introduces
a series of innovative techniques to enhance the translation quality of
Non-Autoregressive Translation (NAT) models while maintaining a substantial
acceleration in inference speed. We propose fine-tuning Pretrained Multilingual
Language Models (PMLMs) with the CTC loss to train NAT models effectively.
Furthermore, we adopt the MASK insertion scheme for up-sampling instead of
token duplication, and we present an embedding distillation method to further
enhance performance. In our experiments, our model outperforms the baseline
autoregressive model (Transformer \textit{base}) on multiple datasets,
including WMT'14 DE$\leftrightarrow$EN, WMT'16 RO$\leftrightarrow$EN, and
IWSLT'14 DE$\leftrightarrow$EN. Notably, our model achieves better performance
than the baseline autoregressive model on the IWSLT'14 En$\leftrightarrow$De
and WMT'16 En$\leftrightarrow$Ro datasets, even without using distillation data
during training. It is worth highlighting that on the IWSLT'14
DE$\rightarrow$EN dataset, our model achieves an impressive BLEU score of
39.59, setting a new state-of-the-art performance. Additionally, our model
exhibits a remarkable speed improvement of 16.35 times compared to the
autoregressive model.

通过使用 CTC loss 进行微调 PMLM 模型、采用 MASK 插入方案进行上采样、使用嵌入蒸馏方法进一步提高性能，使得非自回归模型获得了更好的翻译质量和加速，并在多个数据集上优于自回归模型

使用预训练语言模型、嵌入蒸馏和上采样策略改善 CTC 非自回归翻译质量

Improving Non-autoregressive Translation Quality with Pretrained  Language Model, Embedding Distillation and Upsampling Strategy for CTC

Class-Incremental Learning (CIL) aims to solve the neural networks'
catastrophic forgetting problem, which refers to the fact that once the network
updates on a new task, its performance on previously-learned tasks drops
dramatically. Most successful CIL methods incrementally train a feature
extractor with the aid of stored exemplars, or estimate the feature
distribution with the stored prototypes. However, the stored exemplars would
violate the data privacy concerns, while the stored prototypes might not
reasonably be consistent with a proper feature distribution, hindering the
exploration of real-world CIL applications. In this paper, we propose a method
of \textit{e}mbedding distillation and \textit{Ta}sk-oriented
\textit{g}eneration (\textit{eTag}) for CIL, which requires neither the
exemplar nor the prototype. Instead, eTag achieves a data-free manner to train
the neural networks incrementally. To prevent the feature extractor from
forgetting, eTag distills the embeddings of the network's intermediate blocks.
Additionally, eTag enables a generative network to produce suitable features,
fitting the needs of the top incremental classifier. Experimental results
confirmed that our proposed eTag considerably outperforms the state-of-the-art
methods on CIFAR-100 and ImageNet-sub\footnote{Our code is available in the
Supplementary Materials.

本研究提出了一种称为 eTag 的方法，通过嵌入蒸馏和面向任务的生成，无需存储样本或原型即可增量训练神经网络，以解决 Class-Incremental Learning 中的遗忘问题，并在 CIFAR-100 和 ImageNet-sub 数据集上获得了显著优于现有方法的结果。