Masked auto-encoder pre-training has emerged as a prevalent technique for
initializing and enhancing dense retrieval systems. It generally utilizes
additional Transformer decoder blocks to provide sustainable supervision
signals and compress contextual information into dense representations.
However, the underlying reasons for the effectiveness of such a pre-training
technique remain unclear. The usage of additional Transformer-based decoders
also incurs significant computational costs. In this study, we aim to shed
light on this issue by revealing that masked auto-encoder (MAE) pre-training
with enhanced decoding significantly improves the term coverage of input tokens
in dense representations, compared to vanilla BERT checkpoints. Building upon
this observation, we propose a modification to the traditional MAE by replacing
the decoder of a masked auto-encoder with a completely simplified Bag-of-Word
prediction task. This modification enables the efficient compression of lexical
signals into dense representations through unsupervised pre-training.
Remarkably, our proposed method achieves state-of-the-art retrieval performance
on several large-scale retrieval benchmarks without requiring any additional
parameters, which provides a 67% training speed-up compared to standard masked
auto-encoder pre-training with enhanced decoding.

通过使用增强解码的遮蔽自动编码器预训练，显著提高了稠密表示中输入标记的术语覆盖，从而实现了在大规模检索基准上的最先进的检索性能，无需任何额外参数，相比于使用增强解码的标准遮蔽自动编码器预训练速度提高了 67%。