Transformer-based language models (LMs) are powerful and widely-applicable
tools, but their usefulness is constrained by a finite context window and the
expensive computational cost of processing long text documents. We propose to
adapt pre-trained LMs into AutoCompressors. These models are capable of
compressing long contexts into compact summary vectors, which are then
accessible to the model as soft prompts. Summary vectors are trained with an
unsupervised objective, whereby long documents are processed in segments and
summary vectors from all previous segments are used in language modeling. We
fine-tune OPT models on sequences of up to 30,720 tokens and show that
AutoCompressors can utilize long contexts to improve perplexity. We evaluate
AutoCompressors on in-context learning by compressing task demonstrations. We
find that summary vectors are good substitutes for plain-text demonstrations,
increasing accuracy while reducing inference cost. Finally, we explore the
benefits of pre-computing summary vectors for large corpora by applying summary
vectors to retrieval-augmented language modeling. Overall, AutoCompressors
emerge as a simple and inexpensive solution for extending the context window of
LMs while speeding up inference over long contexts.

该论文旨在介绍一种新型的 AutoCompressors 语言模型，可以将长文档内容压缩成简洁的总结向量，并将其用作语言模型的轻提示，从而提高了模型处理长文本时间的效率，并在任务演示、检索等领域取得了显著进展。