We carry out a comprehensive evaluation of 13 recent models for ranking of
long documents using two popular collections (MS MARCO documents and Robust04).
Our model zoo includes two specialized Transformer models (such as Longformer)
that can process long documents without the need to split them. Along the way,
we document several difficulties regarding training and comparing such models.
Somewhat surprisingly, we find the simple FirstP baseline (truncating documents
to satisfy the input-sequence constraint of a typical Transformer model) to be
quite effective. We analyze the distribution of relevant passages (inside
documents) to explain this phenomenon. We further argue that, despite their
widespread use, Robust04 and MS MARCO documents are not particularly useful for
benchmarking of long-document models.

本研究综合评估了 13 个模型在排名长文档方面的功能，包括两种专用 Transformer 模型，同时分析了训练和比较这些模型的几个困难。结果表明，虽然 Robust04 和 MS MARCO 文件经常被用于长文件模型的基准测试，但我们认为它们并不特别有用。特别是在考虑到 relevant passages 的分布时，Simple FirstP Baseline 很有效。