Multi-document summarization (MDS) is a difficult task in Natural Language
Processing, aiming to summarize information from several documents. However,
the source documents are often insufficient to obtain a qualitative summary. We
propose a retriever-guided model combined with non-parametric memory for
summary generation. This model retrieves relevant candidates from a database
and then generates the summary considering the candidates with a copy mechanism
and the source documents. The retriever is implemented with Approximate Nearest
Neighbor Search (ANN) to search large databases. Our method is evaluated on the
MultiXScience dataset which includes scientific articles. Finally, we discuss
our results and possible directions for future work.

我们提出了一种用于摘要生成的检索引导模型，结合了非参数记忆。该模型使用近似最近邻搜索从数据库中检索相关的候选文本，然后利用复制机制和源文档生成摘要。我们在包括科学文章的 MultiXScience 数据集上评估了我们的方法，并讨论了我们的结果和未来工作的可能方向。

非参数记忆引导的多文档摘要

Non-Parametric Memory Guidance for Multi-Document Summarization

As demonstrated by GPT-3 and T5, transformers grow in capability as parameter
spaces become larger and larger. However, for tasks that require a large amount
of knowledge, non-parametric memory allows models to grow dramatically with a
sub-linear increase in computational cost and GPU memory requirements. Recent
models such as RAG and REALM have introduced retrieval into conditional
generation. These models incorporate neural initial retrieval from a corpus of
passages. We build on this line of research, proposing Re2G, which combines
both neural initial retrieval and reranking into a BART-based
sequence-to-sequence generation. Our reranking approach also permits merging
retrieval results from sources with incomparable scores, enabling an ensemble
of BM25 and neural initial retrieval. To train our system end-to-end, we
introduce a novel variation of knowledge distillation to train the initial
retrieval, reranker, and generation using only ground truth on the target
sequence output. We find large gains in four diverse tasks: zero-shot slot
filling, question answering, fact-checking, and dialog, with relative gains of
9% to 34% over the previous state-of-the-art on the KILT leaderboard. We make
our code available as open source at
this https URL.

Re2G 利用神经检索和排名策略结合 BART 序列生成器，在零 - shot 填槽、问答、事实核查和对话等任务中取得了相对增益为 9% 至 34% 的较大提升，是目前 KILT 排行榜上最先进的模型。

Re2G: 检索、排序、生成

Re2G: Retrieve, Rerank, Generate

Large pre-trained language models have been shown to store factual knowledge
in their parameters, and achieve state-of-the-art results when fine-tuned on
downstream NLP tasks. However, their ability to access and precisely manipulate
knowledge is still limited, and hence on knowledge-intensive tasks, their
performance lags behind task-specific architectures. Additionally, providing
provenance for their decisions and updating their world knowledge remain open
research problems. Pre-trained models with a differentiable access mechanism to
explicit non-parametric memory can overcome this issue, but have so far been
only investigated for extractive downstream tasks. We explore a general-purpose
fine-tuning recipe for retrieval-augmented generation (RAG) -- models which
combine pre-trained parametric and non-parametric memory for language
generation. We introduce RAG models where the parametric memory is a
pre-trained seq2seq model and the non-parametric memory is a dense vector index
of Wikipedia, accessed with a pre-trained neural retriever. We compare two RAG
formulations, one which conditions on the same retrieved passages across the
whole generated sequence, the other can use different passages per token. We
fine-tune and evaluate our models on a wide range of knowledge-intensive NLP
tasks and set the state-of-the-art on three open domain QA tasks, outperforming
parametric seq2seq models and task-specific retrieve-and-extract architectures.
For language generation tasks, we find that RAG models generate more specific,
diverse and factual language than a state-of-the-art parametric-only seq2seq
baseline.

探索一种以维基百科作为参数内存和通过神经检索器访问的显式非参数内存的 pre-trained 具有差异访问机制的 RAG 模型，它可用于在广泛的与知识有关的 NLP 任务中取得 state-of-the-art 的表现。