To extract answers from a large corpus, open-domain question answering (QA)
systems usually rely on information retrieval (IR) techniques to narrow the
search space. Standard inverted index methods such as TF-IDF are commonly used
as thanks to their efficiency. However, their retrieval performance is limited
as they simply use shallow and sparse lexical features. To break the IR
bottleneck, recent studies show that stronger retrieval performance can be
achieved by pretraining a effective paragraph encoder that index paragraphs
into dense vectors. Once trained, the corpus can be pre-encoded into
low-dimensional vectors and stored within an index structure where the
retrieval can be efficiently implemented as maximum inner product search.
Despite the promising results, pretraining such a dense index is expensive
and often requires a very large batch size. In this work, we propose a simple
and resource-efficient method to pretrain the paragraph encoder. First, instead
of using heuristically created pseudo question-paragraph pairs for pretraining,
we utilize an existing pretrained sequence-to-sequence model to build a strong
question generator that creates high-quality pretraining data. Second, we
propose a progressive pretraining algorithm to ensure the existence of
effective negative samples in each batch. Across three datasets, our method
outperforms an existing dense retrieval method that uses 7 times more
computational resources for pretraining.

本研究提出了一种简单且资源有效的方法，利用现有的预训练序列到序列模型来建立强大的问题生成器，并通过渐进式预训练算法确保每个批次有效的负样本存在，以预训练段落编码器。通过三个数据集的实验，本方法在使用 7 倍的计算资源进行预训练的现有浓缩检索方法方面取得了更好的效果。