As pretrained transformer language models continue to achieve
state-of-the-art performance, the Natural Language Processing community has
pushed for advances in model compression and efficient attention mechanisms to
address high computational requirements and limited input sequence length.
Despite these separate efforts, no investigation has been done into the
intersection of these two fields. In this work, we provide an evaluation of
model compression via knowledge distillation on efficient attention
transformers. We provide cost-performance trade-offs for the compression of
state-of-the-art efficient attention architectures and the gains made in
performance in comparison to their full attention counterparts. Furthermore, we
introduce a new long-context Named Entity Recognition dataset, GONERD, to train
and test the performance of NER models on long sequences. We find that
distilled efficient attention transformers can preserve a significant amount of
original model performance, preserving up to 98.6% across short-context tasks
(GLUE, SQUAD, CoNLL-2003), up to 94.6% across long-context
Question-and-Answering tasks (HotpotQA, TriviaQA), and up to 98.8% on
long-context Named Entity Recognition (GONERD), while decreasing inference
times by up to 57.8%. We find that, for most models on most tasks, performing
knowledge distillation is an effective method to yield high-performing
efficient attention models with low costs.

对高效注意力转换模型进行知识蒸馏的模型压缩评估，并通过新的长上下文命名实体识别数据集 GONERD 验证了高效注意力转换模型在保持原始模型性能的同时降低推理时间的效果。