Long text understanding is important yet challenging for natural language
processing. A long article or document usually contains many redundant words
that are not pertinent to its gist and sometimes can be regarded as noise. With
recent advances of abstractive summarization, we propose our \emph{Gist
Detector} to leverage the gist detection ability of a summarization model and
integrate the extracted gist into downstream models to enhance their long text
understanding ability. Specifically, Gist Detector first learns the gist
detection knowledge distilled from a summarization model, and then produces
gist-aware representations to augment downstream models. We evaluate our method
on three different tasks: long document classification, distantly supervised
open-domain question answering, and non-parallel text style transfer. The
experimental results show that our method can significantly improve the
performance of baseline models on all tasks.

通过使用抽象化摘要生成模型的主旨检测能力，我们提出了 Gist Detector 来加强下游模型对长文本的理解能力，并评估了该方法在长文档分类、远程监督的开放领域问答以及非平行文本风格转换等三个不同任务上的性能表现，结果显示我们的方法在所有任务上都显著提升了基线模型的性能。

利用从摘要模型蒸馏的知识提升长文本理解能力

Improving Long Text Understanding with Knowledge Distilled from  Summarization Model

Large language models (LLMs) have achieved impressive performance in numerous
domains but often struggle to process lengthy inputs effectively and
efficiently due to limited length generalization and attention's quadratic
computational demands. Many sought to mitigate this by restricting the
attention window within the pre-trained length. However, these methods
introduce new issues such as ignoring the middle context and requiring
additional training. To address these problems, we propose LongHeads, a
training-free framework that enhances LLM's long context ability by unlocking
multi-head attention's untapped potential. Instead of allowing each head to
attend to the full sentence, which struggles with generalizing to longer
sequences due to out-of-distribution (OOD) issues, we allow each head to
process in-distribution length by selecting and attending to important context
chunks. To this end, we propose a chunk selection strategy that relies on the
inherent correlation between the query and the key representations, efficiently
distributing context chunks to different heads. In this way, each head ensures
it can effectively process attended tokens within the trained length, while
different heads in different layers can collectively process longer contexts.
LongHeads works efficiently in linear time, fits seamlessly with many LLMs that
use relative positional encoding. Our extensive empirical analyses verify
LongHeads's efficacy in extending the usable context window for existing
models, showcasing its promise for enhancing long text understanding.

通过解锁多头注意力的潜力，我们提出了一个无需额外训练的框架 LongHeads，以增强大语言模型（LLMs）在处理长篇输入方面的能力，通过选择和关注重要的上下文块来确保各个头能够有效地处理训练长度内的被关注的标记，同时不同层的不同头可以共同处理更长的上下文，有效地扩展了现有模型的可用上下文窗口，展示了对增强长文本理解的潜力。