Retrieving documents and prepending them in-context at inference time
improves performance of language model (LMs) on a wide range of tasks. However,
these documents, often spanning hundreds of words, make inference substantially
more expensive. We propose compressing the retrieved documents into textual
summaries prior to in-context integration. This not only reduces the
computational costs but also relieves the burden of LMs to identify relevant
information in long retrieved documents. We present two compressors -- an
extractive compressor which selects useful sentences from retrieved documents
and an abstractive compressor which generates summaries by synthesizing
information from multiple documents. Both compressors are trained to improve
LMs' performance on end tasks when the generated summaries are prepended to the
LMs' input, while keeping the summary concise.If the retrieved documents are
irrelevant to the input or offer no additional information to LM, our
compressor can return an empty string, implementing selective augmentation.We
evaluate our approach on language modeling task and open domain question
answering task. We achieve a compression rate of as low as 6% with minimal loss
in performance for both tasks, significantly outperforming the off-the-shelf
summarization models. We show that our compressors trained for one LM can
transfer to other LMs on the language modeling task and provide summaries
largely faithful to the retrieved documents.

将检索到的文档压缩为文本摘要，以降低计算成本并减轻语言模型的负担，以提高任务性能；在语言建模和开放域问题回答任务中，以最低为 6% 的压缩率实现性能上最小损失，并在给定一种语言模型训练方法时能够迁移到其他语言模型，并提供与检索文档基本一致的摘要。

RECOMP: 通过压缩和选择性增强改进基于检索的语言模型

RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective  Augmentation

While Wikipedia exists in 287 languages, its content is unevenly distributed
among them. In this work, we investigate the generation of open domain
Wikipedia summaries in underserved languages using structured data from
Wikidata. To this end, we propose a neural network architecture equipped with
copy actions that learns to generate single-sentence and comprehensible textual
summaries from Wikidata triples. We demonstrate the effectiveness of the
proposed approach by evaluating it against a set of baselines on two languages
of different natures: Arabic, a morphological rich language with a larger
vocabulary than English, and Esperanto, a constructed language known for its
easy acquisition.

本文探讨如何使用 Wikidata 中的结构化数据来生成在 Wikipedia 贫穷语言中的开放领域概要。 为此，作者提出了一种神经网络结构，该结构配备有复制操作，可以学习从 Wikidata 三元组中生成单个句子和可理解的文本摘要。作者在阿拉伯语和世界语这两种不同特性的语言上对所提出的方法进行了评估，并证明了它的有效性。