Despite pre-training's progress in many important NLP tasks, it remains to
explore effective pre-training strategies for dense retrieval. In this paper,
we propose RetroMAE, a new retrieval oriented pre-training paradigm based on
Masked Auto-Encoder (MAE). RetroMAE is highlighted by three critical designs.
1) A novel MAE workflow, where the input sentence is polluted for encoder and
decoder with different masks. The sentence embedding is generated from the
encoder's masked input; then, the original sentence is recovered based on the
sentence embedding and the decoder's masked input via masked language modeling.
2) Asymmetric model structure, with a full-scale BERT like transformer as
encoder, and a one-layer transformer as decoder. 3) Asymmetric masking ratios,
with a moderate ratio for encoder: 15~30%, and an aggressive ratio for decoder:
50~70%. Our framework is simple to realize and empirically competitive: the
pre-trained models dramatically improve the SOTA performances on a wide range
of dense retrieval benchmarks, like BEIR and MS MARCO. The source code and
pre-trained models are made publicly available at
this https URL so as to inspire more interesting
research.

本研究提出一种新的基于 Masked Auto-Encoder 的检索导向的预训练模型 RetroMAE，通过三个关键设计，包括污染输入、不对称模型结构和不对称的遮盖比率，取得了令人满意的实验结果，大幅提高了在 BEIR 和 MS MARCO 等检索基准测试中的性能表现。