We study ranked list truncation (RLT) from a novel "retrieve-then-re-rank"
perspective, where we optimize re-ranking by truncating the retrieved list
(i.e., trim re-ranking candidates). RLT is crucial for re-ranking as it can
improve re-ranking efficiency by sending variable-length candidate lists to a
re-ranker on a per-query basis. It also has the potential to improve re-ranking
effectiveness. Despite its importance, there is limited research into applying
RLT methods to this new perspective. To address this research gap, we reproduce
existing RLT methods in the context of re-ranking, especially newly emerged
large language model (LLM)-based re-ranking. In particular, we examine to what
extent established findings on RLT for retrieval are generalizable to the
"retrieve-then-re-rank" setup from three perspectives: (i) assessing RLT
methods in the context of LLM-based re-ranking with lexical first-stage
retrieval, (ii) investigating the impact of different types of first-stage
retrievers on RLT methods, and (iii) investigating the impact of different
types of re-rankers on RLT methods. We perform experiments on the TREC 2019 and
2020 deep learning tracks, investigating 8 RLT methods for pipelines involving
3 retrievers and 2 re-rankers. We reach new insights into RLT methods in the
context of re-ranking.

我们从新的 “检索 - 然后重新排名” 的角度研究了排名列表截取（RLT），通过对检索列表进行截断（即修剪重新排名的候选项），我们优化了重新排名。我们通过复现现有的 RLT 方法来填补这个研究空白，特别是基于大型语言模型（LLM）的重新排名。我们从三个角度考察了在 “检索 - 然后重新排名” 设置下针对 LLM 重新排名的 RLT 方法，分别是评估 LLM 基于词汇的第一阶段检索下的 RLT 方法、研究不同类型的第一阶段检索器对 RLT 方法的影响以及研究不同类型的重新排名器对 RLT 方法的影响。我们在 TREC 2019 和 2020 深度学习环节上进行实验，研究 8 种 RLT 方法在涉及 3 个检索器和 2 个重新排名器的流水线上的表现，并对 RLT 方法在重新排名的背景下得出了新的见解。

基于大型语言模型的排序列表截断

Ranked List Truncation for Large Language Model-based Re-Ranking

Work in information retrieval has traditionally focused on ranking and
relevance: given a query, return some number of results ordered by relevance to
the user. However, the problem of determining how many results to return, i.e.
how to optimally truncate the ranked result list, has received less attention
despite being of critical importance in a range of applications. Such
truncation is a balancing act between the overall relevance, or usefulness of
the results, with the user cost of processing more results. In this work, we
propose Choppy, an assumption-free model based on the widely successful
Transformer architecture, to the ranked list truncation problem. Needing
nothing more than the relevance scores of the results, the model uses a
powerful multi-head attention mechanism to directly optimize any user-defined
IR metric. We show Choppy improves upon recent state-of-the-art methods.

该研究提出了一种基于 Transformer 网络的模型 Choppy，利用多头注意力机制来优化用户定义的 IR 度量标准，解决排名结果列表截断问题，取得了优于最近现有的最佳方法的效果。