Dense passage retrieval aims to retrieve the relevant passages of a query
from a large corpus based on dense representations (i.e., vectors) of the query
and the passages. Recent studies have explored improving pre-trained language
models to boost dense retrieval performance. This paper proposes CoT-MAE
(ConTextual Masked Auto-Encoder), a simple yet effective generative
pre-training method for dense passage retrieval. CoT-MAE employs an asymmetric
encoder-decoder architecture that learns to compress the sentence semantics
into a dense vector through self-supervised and context-supervised masked
auto-encoding. Precisely, self-supervised masked auto-encoding learns to model
the semantics of the tokens inside a text span, and context-supervised masked
auto-encoding learns to model the semantical correlation between the text
spans. We conduct experiments on large-scale passage retrieval benchmarks and
show considerable improvements over strong baselines, demonstrating the high
efficiency of CoT-MAE. Our code is available at
this https URL.

本研究提出 CoT-MAE 方法，通过自监督和上下文监督的遮蔽自编码学习将句子语义压缩成密集向量，进而在大规模的检索基准数据集上实现了相比强基线更高效的检索表现。