Passage retrieval aims to retrieve relevant passages from large collections
of the open-domain corpus. Contextual Masked Auto-Encoding has been proven
effective in representation bottleneck pre-training of a monolithic
dual-encoder for passage retrieval. Siamese or fully separated dual-encoders
are often adopted as basic retrieval architecture in the pre-training and
fine-tuning stages for encoding queries and passages into their latent
embedding spaces. However, simply sharing or separating the parameters of the
dual-encoder results in an imbalanced discrimination of the embedding spaces.
In this work, we propose to pre-train Contextual Masked Auto-Encoder with
Mixture-of-Textual-Experts (CoT-MoTE). Specifically, we incorporate
textual-specific experts for individually encoding the distinct properties of
queries and passages. Meanwhile, a shared self-attention layer is still kept
for unified attention modeling. Results on large-scale passage retrieval
benchmarks show steady improvement in retrieval performances. The quantitive
analysis also shows a more balanced discrimination of the latent embedding
spaces.

该研究探讨了使用 Mixture-of-Textual-Experts 的 Contextual Masked Auto-Encoding 预训练模型进行 Passage retrieval 以提高检索效果。实验结果显示该方法可以更加平衡地判别潜在的嵌入空间，并取得了稳定的检索性能提升。

CoT-MoTE: 通过文本专家混合探索上下文掩码自编码器预训练以进行段落检索

CoT-MoTE: Exploring ConTextual Masked Auto-Encoder Pre-training with  Mixture-of-Textual-Experts for Passage Retrieval

Multi-label classification is the challenging task of predicting the presence
and absence of multiple targets, involving representation learning and label
correlation modeling. We propose a novel framework for multi-label
classification, Multivariate Probit Variational AutoEncoder (MPVAE), that
effectively learns latent embedding spaces as well as label correlations. MPVAE
learns and aligns two probabilistic embedding spaces for labels and features
respectively. The decoder of MPVAE takes in the samples from the embedding
spaces and models the joint distribution of output targets under a Multivariate
Probit model by learning a shared covariance matrix. We show that MPVAE
outperforms the existing state-of-the-art methods on a variety of application
domains, using public real-world datasets. MPVAE is further shown to remain
robust under noisy settings. Lastly, we demonstrate the interpretability of the
learned covariance by a case study on a bird observation dataset.

本篇论文提出了一个用于多标签分类的新型框架，即多元 Probit 变分自编码器（MPVAE），该框架通过有效地学习潜在的嵌入空间和标签之间的相关性来实现。同时，MPVAE 在公开真实世界数据集上优于现有的最先进方法，并且进一步证明了在嘈杂环境下的稳健性。最后，通过鸟类观测数据集的案例研究，展示了学习共享协方差矩阵的可解释性。