This study investigates the existence of positional biases in
Transformer-based models for text representation learning, particularly in the
context of web document retrieval. We build on previous research that
demonstrated loss of information in the middle of input sequences for causal
language models, extending it to the domain of representation learning. We
examine positional biases at various stages of training for an encoder-decoder
model, including language model pre-training, contrastive pre-training, and
contrastive fine-tuning. Experiments with the MS-MARCO document collection
reveal that after contrastive pre-training the model already generates
embeddings that better capture early contents of the input, with fine-tuning
further aggravating this effect.

这项研究调查了 Transformer-based 模型在文本表征学习中的位置偏差存在情况，特别是在网络文档检索的背景下。我们基于先前研究的基础上，拓展了关于因果性语言模型输入序列中信息丢失的问题，将其应用于表征学习领域。我们在编码器 - 解码器模型的各个训练阶段，包括语言模型预训练、对比度预训练和对比度微调中，研究了位置偏差。通过对 MS-MARCO 文档集的实验发现，在对比度预训练之后，该模型已经生成了能够更好地捕捉输入早期内容的嵌入向量，而微调进一步加剧了这种效果。

驻足于起始处：语言模型如何嵌入长篇文档以进行稠密检索

Dwell in the Beginning: How Language Models Embed Long Documents for  Dense Retrieval

This paper develops a model that addresses sentence embedding, a hot topic in
current natural language processing research, using recurrent neural networks
with Long Short-Term Memory (LSTM) cells. Due to its ability to capture long
term memory, the LSTM-RNN accumulates increasingly richer information as it
goes through the sentence, and when it reaches the last word, the hidden layer
of the network provides a semantic representation of the whole sentence. In
this paper, the LSTM-RNN is trained in a weakly supervised manner on user
click-through data logged by a commercial web search engine. Visualization and
analysis are performed to understand how the embedding process works. The model
is found to automatically attenuate the unimportant words and detects the
salient keywords in the sentence. Furthermore, these detected keywords are
found to automatically activate different cells of the LSTM-RNN, where words
belonging to a similar topic activate the same cell. As a semantic
representation of the sentence, the embedding vector can be used in many
different applications. These automatic keyword detection and topic allocation
abilities enabled by the LSTM-RNN allow the network to perform document
retrieval, a difficult language processing task, where the similarity between
the query and documents can be measured by the distance between their
corresponding sentence embedding vectors computed by the LSTM-RNN. On a web
search task, the LSTM-RNN embedding is shown to significantly outperform
several existing state of the art methods. We emphasize that the proposed model
generates sentence embedding vectors that are specially useful for web document
retrieval tasks. A comparison with a well known general sentence embedding
method, the Paragraph Vector, is performed. The results show that the proposed
method in this paper significantly outperforms it for web document retrieval
task.

本文提出了一种基于循环神经网络和长短时记忆单元的句子嵌入模型，该模型能够自动侦测句子中关键词和主题，从而进行网络文档检索等艰难的语言处理任务，并在性能上显著优于现有的先进方法。