Passage retrieval aims to retrieve relevant passages from large collections
of the open-domain corpus. Contextual Masked Auto-Encoding has been proven
effective in representation bottleneck pre-training of a monolithic
dual-encoder for passage retrieval. Siamese or fully separated dual-encoders
are often adopted as basic retrieval architecture in the pre-training and
fine-tuning stages for encoding queries and passages into their latent
embedding spaces. However, simply sharing or separating the parameters of the
dual-encoder results in an imbalanced discrimination of the embedding spaces.
In this work, we propose to pre-train Contextual Masked Auto-Encoder with
Mixture-of-Textual-Experts (CoT-MoTE). Specifically, we incorporate
textual-specific experts for individually encoding the distinct properties of
queries and passages. Meanwhile, a shared self-attention layer is still kept
for unified attention modeling. Results on large-scale passage retrieval
benchmarks show steady improvement in retrieval performances. The quantitive
analysis also shows a more balanced discrimination of the latent embedding
spaces.

该研究探讨了使用 Mixture-of-Textual-Experts 的 Contextual Masked Auto-Encoding 预训练模型进行 Passage retrieval 以提高检索效果。实验结果显示该方法可以更加平衡地判别潜在的嵌入空间，并取得了稳定的检索性能提升。