In this paper, we propose SimLM (Similarity matching with Language Model
pre-training), a simple yet effective pre-training method for dense passage
retrieval. It employs a simple bottleneck architecture that learns to compress
the passage information into a dense vector through self-supervised
pre-training. We use a replaced language modeling objective, which is inspired
by ELECTRA, to improve the sample efficiency and reduce the mismatch of the
input distribution between pre-training and fine-tuning. SimLM only requires
access to unlabeled corpus, and is more broadly applicable when there are no
labeled data or queries. We conduct experiments on several large-scale passage
retrieval datasets, and show substantial improvements over strong baselines
under various settings. Remarkably, SimLM even outperforms multi-vector
approaches such as ColBERTv2 which incurs significantly more storage cost.

本文提出一种简单而有效的预训练方法 SimLM，用于针对稠密段落检索，其使用简单的瓶颈架构通过自监督预训练学习将段落信息压缩成一种密集向量，并使用替换的语言建模目标，该目标灵感来自 ELECTRA，以提高样本效率并减少预训练和微调之间输入分布的差异。SimLM 只需要访问未标记的语料库，在没有标记数据或查询时更普遍适用。在多个大规模段落检索数据集上进行实验证明，在多种设置下，SimLM 都比强基线实现了实质性的改进。显著的是，SimLM 甚至胜过了多向量方法，如 ColBERTv2，这需要更高的存储成本。