To better support information retrieval tasks such as web search and
open-domain question answering, growing effort is made to develop
retrieval-oriented language models, e.g., RetroMAE and many others. Most of the
existing works focus on improving the semantic representation capability for
the contextualized embedding of the [CLS] token. However, recent study shows
that the ordinary tokens besides [CLS] may provide extra information, which
help to produce a better representation effect. As such, it's necessary to
extend the current methods where all contextualized embeddings can be jointly
pre-trained for the retrieval tasks. In this work, we propose a novel
pre-training method called Duplex Masked Auto-Encoder, a.k.a. DupMAE. It is
designed to improve the quality of semantic representation where all
contextualized embeddings of the pre-trained model can be leveraged. It takes
advantage of two complementary auto-encoding tasks: one reconstructs the input
sentence on top of the [CLS] embedding; the other one predicts the bag-of-words
feature of the input sentence based on the ordinary tokens' embeddings. The two
tasks are jointly conducted to train a unified encoder, where the whole
contextualized embeddings are aggregated in a compact way to produce the final
semantic representation. DupMAE is simple but empirically competitive: it
substantially improves the pre-trained model's representation capability and
transferability, where superior retrieval performances can be achieved on
popular benchmarks, like MS MARCO and BEIR.

本文提出了一种新的预训练方法，名为 Duplex Masked Auto-Encoder（DupMAE），以改善用于信息检索任务的上下文嵌入的语义表示质量，其中所有上下文嵌入都可以被利用。通过两个互补的自编码任务，联合训练一个统一的编码器，实现所有上下文化嵌入的紧凑汇总，生成最终的语义表示。

RetroMAE-2: 双工遮蔽自编码器用于预训练面向检索的语言模型

RetroMAE-2: Duplex Masked Auto-Encoder For Pre-Training  Retrieval-Oriented Language Models

To better support retrieval applications such as web search and question
answering, growing effort is made to develop retrieval-oriented language
models. Most of the existing works focus on improving the semantic
representation capability for the contextualized embedding of [CLS] token.
However, recent study shows that the ordinary tokens besides [CLS] may provide
extra information, which helps to produce a better representation effect. As
such, it's necessary to extend the current methods where all contextualized
embeddings can be jointly pre-trained for the retrieval tasks.
With this motivation, we propose a new pre-training method: duplex masked
auto-encoder, a.k.a. DupMAE, which targets on improving the semantic
representation capacity for the contextualized embeddings of both [CLS] and
ordinary tokens. It introduces two decoding tasks: one is to reconstruct the
original input sentence based on the [CLS] embedding, the other one is to
minimize the bag-of-words loss (BoW) about the input sentence based on the
entire ordinary tokens' embeddings. The two decoding losses are added up to
train a unified encoding model. The embeddings from [CLS] and ordinary tokens,
after dimension reduction and aggregation, are concatenated as one unified
semantic representation for the input. DupMAE is simple but empirically
competitive: with a small decoding cost, it substantially contributes to the
model's representation capability and transferability, where remarkable
improvements are achieved on MS MARCO and BEIR benchmarks.

本文介绍了一种新的预训练方法 ——DupMAE，该方法可同时提高 [CLS] 和普通 token 的语境嵌入的语义表示能力，取得了在 MS MARCO 和 BEIR 基准测试中显著的改进效果。