Growing techniques have been emerging to improve the performance of passage
retrieval. As an effective representation bottleneck pretraining technique, the
contextual masked auto-encoder utilizes contextual embedding to assist in the
reconstruction of passages. However, it only uses a single auto-encoding
pre-task for dense representation pre-training. This study brings multi-view
modeling to the contextual masked auto-encoder. Firstly, multi-view
representation utilizes both dense and sparse vectors as multi-view
representations, aiming to capture sentence semantics from different aspects.
Moreover, multiview decoding paradigm utilizes both autoencoding and
auto-regressive decoders in representation bottleneck pre-training, aiming to
provide both reconstructive and generative signals for better contextual
representation pretraining. We refer to this multi-view pretraining method as
CoT-MAE v2. Through extensive experiments, we show that CoT-MAE v2 is effective
and robust on large-scale passage retrieval benchmarks and out-of-domain
zero-shot benchmarks.

通过多视图模型和上下文遮掩自编码器的多视图预训练方法，该研究旨在提供更好的上下文表示预训练，并在大规模文本检索基准和领域外零 - shot 基准测试中表现出有效性和鲁棒性。