Recent research demonstrates the effectiveness of using fine-tuned language
models~(LM) for dense retrieval. However, dense retrievers are hard to train,
typically requiring heavily engineered fine-tuning pipelines to realize their
full potential. In this paper, we identify and address two underlying problems
of dense retrievers: i)~fragility to training data noise and ii)~requiring
large batches to robustly learn the embedding space. We use the recently
proposed Condenser pre-training architecture, which learns to condense
information into the dense vector through LM pre-training. On top of it, we
propose coCondenser, which adds an unsupervised corpus-level contrastive loss
to warm up the passage embedding space. Retrieval experiments on MS-MARCO,
Natural Question, and Trivia QA datasets show that coCondenser removes the need
for heavy data engineering such as augmentation, synthesis, or filtering, as
well as the need for large batch training. It shows comparable performance to
RocketQA, a state-of-the-art, heavily engineered system, using simple small
batch fine-tuning.

本文提出了 coCondenser 方法，将 Condenser 预训练架构与无监督语料级对比损失相结合，减轻了密集检索器对大规模训练数据和数据工程的依赖，并通过实验表明，该方法具有与 RocketQA 相当的性能。