Knowledge-intensive language tasks (KILTs) typically require retrieving
relevant documents from trustworthy corpora, e.g., Wikipedia, to produce
specific answers. Very recently, a pre-trained generative retrieval model for
KILTs, named CorpusBrain, was proposed and reached new state-of-the-art
retrieval performance. However, most existing research on KILTs, including
CorpusBrain, has predominantly focused on a static document collection,
overlooking the dynamic nature of real-world scenarios, where new documents are
continuously being incorporated into the source corpus. To address this gap, it
is crucial to explore the capability of retrieval models to effectively handle
the dynamic retrieval scenario inherent in KILTs.
In this work, we first introduce the continual document learning (CDL) task
for KILTs and build a novel benchmark dataset named KILT++ based on the
original KILT dataset for evaluation. Then, we conduct a comprehensive study
over the use of pre-trained CorpusBrain on KILT++. Unlike the promising results
in the stationary scenario, CorpusBrain is prone to catastrophic forgetting in
the dynamic scenario, hence hampering the retrieval performance. To alleviate
this issue, we propose CorpusBrain++, a continual generative pre-training
framework. Empirical results demonstrate the significant effectiveness and
remarkable efficiency of CorpusBrain++ in comparison to both traditional and
generative IR methods.

通过建立一个名为 KILT++ 的基准数据集，我们对预先训练的 CorpusBrain 在知识密集语言任务中的应用进行了全面研究，发现在动态检索场景中，CorpusBrain 往往容易发生灾难性遗忘，从而影响检索性能，为了缓解这一问题，我们提出了 CorpusBrain++，这是一个持续生成预训练框架，在传统和生成式信息检索方法方面，经实验证明 CorpusBrain++ 具有显著的有效性和卓越的效率。

CorpusBrain++：一种用于知识密集型语言任务的连续生成预训练框架

CorpusBrain++: A Continual Generative Pre-Training Framework for  Knowledge-Intensive Language Tasks

Retrieval-enhanced text generation, which aims to leverage passages retrieved
from a large passage corpus for delivering a proper answer given the input
query, has shown remarkable progress on knowledge-intensive language tasks such
as open-domain question answering and knowledge-enhanced dialogue generation.
However, the retrieved passages are not ideal for guiding answer generation
because of the discrepancy between retrieval and generation, i.e., the
candidate passages are all treated equally during the retrieval procedure
without considering their potential to generate the proper answers. This
discrepancy makes a passage retriever deliver a sub-optimal collection of
candidate passages to generate answers. In this paper, we propose the
GeneRative Knowledge Improved Passage Ranking (GripRank) approach, addressing
the above challenge by distilling knowledge from a generative passage estimator
(GPE) to a passage ranker, where the GPE is a generative language model used to
measure how likely the candidate passages can generate the proper answer. We
realize the distillation procedure by teaching the passage ranker learning to
rank the passages ordered by the GPE. Furthermore, we improve the distillation
quality by devising a curriculum knowledge distillation mechanism, which allows
the knowledge provided by the GPE can be progressively distilled to the ranker
through an easy-to-hard curriculum, enabling the passage ranker to correctly
recognize the provenance of the answer from many plausible candidates. We
conduct extensive experiments on four datasets across three knowledge-intensive
language tasks. Experimental results show advantages over the state-of-the-art
methods for both passage ranking and answer generation on the KILT benchmark.

本文提出了一种名为 GripRank 的算法，它通过从一个生成式的通道评估器到一个通道排名器的知识精华的提炼来解决候选通道在检索过程中被平等对待的问题，并通过设计一套课程知识提炼机制来提升知识的提炼质量，实现了对答案的生成和文本排名的优化，最终在 KILT 基准测试中展现出优于同类方法的结果。

GripRank: 通过生成式知识改进的段落排序填补检索和生成之间的差距

GripRank: Bridging the Gap between Retrieval and Generation via the  Generative Knowledge Improved Passage Ranking

Knowledge-intensive language tasks (KILT) usually require a large body of
information to provide correct answers. A popular paradigm to solve this
problem is to combine a search system with a machine reader, where the former
retrieves supporting evidences and the latter examines them to produce answers.
Recently, the reader component has witnessed significant advances with the help
of large-scale pre-trained generative models. Meanwhile most existing solutions
in the search component rely on the traditional ``index-retrieve-then-rank''
pipeline, which suffers from large memory footprint and difficulty in
end-to-end optimization. Inspired by recent efforts in constructing model-based
IR models, we propose to replace the traditional multi-step search pipeline
with a novel single-step generative model, which can dramatically simplify the
search process and be optimized in an end-to-end manner. We show that a strong
generative retrieval model can be learned with a set of adequately designed
pre-training tasks, and be adopted to improve a variety of downstream KILT
tasks with further fine-tuning. We name the pre-trained generative retrieval
model as CorpusBrain as all information about the corpus is encoded in its
parameters without the need of constructing additional index. Empirical results
show that CorpusBrain can significantly outperform strong baselines for the
retrieval task on the KILT benchmark and establish new state-of-the-art
downstream performances. We also show that CorpusBrain works well under zero-
and low-resource settings.

本文提出了一种基于生成式检索模型的信息检索方法，取代了传统的多步骤索引 - 检索 - 排序流程。该方法能带来显著的简化和端到端的优化，实现了对知识密集语言任务的有效检索和提高，相关检索系统名为 CorpusBrain。