As pretrained transformer language models continue to achieve
state-of-the-art performance, the Natural Language Processing community has
pushed for advances in model compression and efficient attention mechanisms to
address high computational requirements and limited input sequence length.
Despite these separate efforts, no investigation has been done into the
intersection of these two fields. In this work, we provide an evaluation of
model compression via knowledge distillation on efficient attention
transformers. We provide cost-performance trade-offs for the compression of
state-of-the-art efficient attention architectures and the gains made in
performance in comparison to their full attention counterparts. Furthermore, we
introduce a new long-context Named Entity Recognition dataset, GONERD, to train
and test the performance of NER models on long sequences. We find that
distilled efficient attention transformers can preserve a significant amount of
original model performance, preserving up to 98.6% across short-context tasks
(GLUE, SQUAD, CoNLL-2003), up to 94.6% across long-context
Question-and-Answering tasks (HotpotQA, TriviaQA), and up to 98.8% on
long-context Named Entity Recognition (GONERD), while decreasing inference
times by up to 57.8%. We find that, for most models on most tasks, performing
knowledge distillation is an effective method to yield high-performing
efficient attention models with low costs.

对高效注意力转换模型进行知识蒸馏的模型压缩评估，并通过新的长上下文命名实体识别数据集 GONERD 验证了高效注意力转换模型在保持原始模型性能的同时降低推理时间的效果。

高效 Transformer 知识蒸馏：绩效评估

Efficient Transformer Knowledge Distillation: A Performance Review

Random-feature-based attention (RFA) is an efficient approximation of softmax
attention with linear runtime and space complexity. However, the approximation
gap between RFA and conventional softmax attention is not well studied. Built
upon previous progress of RFA, we characterize this gap through the lens of
control variates and show that RFA can be decomposed into a sum of multiple
control variate estimators for each element in the sequence. This new framework
reveals that exact softmax attention can be recovered from RFA by manipulating
each control variate. Besides, it allows us to develop a more flexible form of
control variates, resulting in a novel attention mechanism that significantly
reduces the approximation gap while maintaining linear complexity. Extensive
experiments demonstrate that our model outperforms state-of-the-art efficient
attention mechanisms on both vision and language tasks.

通过控制变量的方法，我们将随机特征关注（RFA）分解成多个控制变量估计器的和，从而揭示了 RFA 和标准 softmax attention 之间的逼近差距。我们开发了一种更灵活的控制变量形式，得到了一种新颖的注意机制，该机制在保持线性复杂度的同时，显著降低了逼近差距，对于视觉和语言任务都比最新的高效注意机制表现更好。