Pre-trained transformers has declared its success in many NLP tasks. One
thread of work focuses on training bi-encoder models (i.e., dense retrievers)
to effectively encode sentences or passages into single-vector dense vectors
for efficient approximate nearest neighbor (ANN) search. However, recent work
has demonstrated that transformers pre-trained with mask language modeling
(MLM) are not capable of effectively aggregating text information into a single
dense vector due to task-mismatch between pre-training and fine-tuning.
Therefore, computationally expensive techniques have been adopted to train
dense retrievers, such as large batch size, knowledge distillation or post
pre-training. In this work, we present a simple approach to effectively
aggregate textual representation from the pre-trained transformer into a dense
vector. Extensive experiments show that our approach improves the robustness of
the single-vector approach under both in-domain and zero-shot evaluations
without any computationally expensive training techniques. Our work
demonstrates that MLM pre-trained transformers can be used to effectively
encode text information into a single-vector for dense retrieval. Code are
available at: this https URL

本研究采用一种简单的方法，有效地将预训练转换器中的文本表示聚合成稠密向量，从而提高了在不同领域和零样本评估下单向量方法的鲁棒性，进而表明 MLM 预训练变形金刚可以用于将文本信息编码成单个向量以进行密集检索。

Aggretriever：一种用于稳健稠密段落检索的简单文本表示聚合方法

Aggretriever: A Simple Approach to Aggregate Textual Representation for  Robust Dense Passage Retrieval

Neural machine translation with millions of parameters is vulnerable to
unfamiliar inputs. We propose Token Drop to improve generalization and avoid
overfitting for the NMT model. Similar to word dropout, whereas we replace
dropped token with a special token instead of setting zero to words. We further
introduce two self-supervised objectives: Replaced Token Detection and Dropped
Token Prediction. Our method aims to force model generating target translation
with less information, in this way the model can learn textual representation
better. Experiments on Chinese-English and English-Romanian benchmark
demonstrate the effectiveness of our approach and our model achieves
significant improvements over a strong Transformer baseline.

通过引入 Token Drop 以及两种自监督目标，提升神经机器翻译的泛化能力和避免过拟合，实验结果表明该方法在中英和英罗马尼亚基准数据集上表现显著优于强 Transformer 基线模型。

神经机器翻译的 Token Drop 机制

Token Drop mechanism for Neural Machine Translation

Classifying a visual concept merely from its associated online textual
source, such as a Wikipedia article, is an attractive research topic in
zero-shot learning because it alleviates the burden of manually collecting
semantic attributes. Several recent works have pursued this approach by
exploring various ways of connecting the visual and text domains. This paper
revisits this idea by stepping further to consider one important factor: the
textual representation is usually too noisy for the zero-shot learning
application. This consideration motivates us to design a simple-but-effective
zero-shot learning method capable of suppressing noise in the text.
More specifically, we propose an $l_{2,1}$-norm based objective function
which can simultaneously suppress the noisy signal in the text and learn a
function to match the text document and visual features. We also develop an
optimization algorithm to efficiently solve the resulting problem. By
conducting experiments on two large datasets, we demonstrate that the proposed
method significantly outperforms the competing methods which rely on online
information sources but without explicit noise suppression. We further make an
in-depth analysis of the proposed method and provide insight as to what kind of
information in documents is useful for zero-shot learning.

该研究探索了基于 Wikipedia 等在线文本源的零样本学习中，文本信息与视觉信息结合的方法，提出了一种基于 $l_{2,1}$ 正则化的目标函数，同时抑制文本中的噪声信号并学习文本信息与视觉特征的匹配，实验证明该方法明显优于其他依赖文本信息但无明确噪声抑制的方法。