Recently, Dense Retrieval (DR) has become a promising solution to document
retrieval, where document representations are used to perform effective and
efficient semantic search. However, DR remains challenging on long documents,
due to the quadratic complexity of its Transformer-based encoder and the finite
capacity of a low-dimension embedding. Current DR models use suboptimal
strategies such as truncating or splitting-and-pooling to long documents
leading to poor utilization of whole document information. In this work, to
tackle this problem, we propose Segment representation learning for long
documents Dense Retrieval (SeDR). In SeDR, Segment-Interaction Transformer is
proposed to encode long documents into document-aware and segment-sensitive
representations, while it holds the complexity of splitting-and-pooling and
outperforms other segment-interaction patterns on DR. Since GPU memory
requirements for long document encoding causes insufficient negatives for DR
training, Late-Cache Negative is further proposed to provide additional cache
negatives for optimizing representation learning. Experiments on MS MARCO and
TREC-DL datasets show that SeDR achieves superior performance among DR models,
and confirm the effectiveness of SeDR on long document retrieval.

通过提出基于区间表征的密集检索技术（SeDR），采用区间交互 Transformer 编码超长的文档，并提出延迟缓存负样本的方法，提高了文档的表征质量和检索性能。在 MS MARCO 和 TREC-DL 数据集上的实验证明了该方法在长文档检索中的优越性。

SeDR: 面向长文档的片段表示学习与密集检索

SeDR: Segment Representation Learning for Long Documents Dense Retrieval

Long Document retrieval (DR) has always been a tremendous challenge for
reading comprehension and information retrieval. The pre-training model has
achieved good results in the retrieval stage and Ranking for long documents in
recent years. However, there is still some crucial problem in long document
ranking, such as data label noises, long document representations, negative
data Unbalanced sampling, etc. To eliminate the noise of labeled data and to be
able to sample the long documents in the search reasonably negatively, we
propose the bag sampling method and the group-wise Localized Contrastive
Estimation(LCE) method. We use the head middle tail passage for the long
document to encode the long document, and in the retrieval, stage Use dense
retrieval to generate the candidate's data. The retrieval data is divided into
multiple bags at the ranking stage, and negative samples are selected in each
bag. After sampling, two losses are combined. The first loss is LCE. To fit bag
sampling well, after query and document are encoded, the global features of
each group are extracted by convolutional layer and max-pooling to improve the
model's resistance to the impact of labeling noise, finally, calculate the LCE
group-wise loss. Notably, our model shows excellent performance on the MS MARCO
Long document ranking leaderboard.

该研究提出了预训练模型、包采样和基于组的本地对比评估（LCE）方法以消除长文档排序中的噪音、数据标签噪音、负数据不平衡采样等问题。在 MS MARCO 长文档排名榜上表现出了良好的性能。