Large Language Models (LLMs) demonstrate impressive effectiveness in
zero-shot document ranking tasks. Pointwise, Pairwise, and Listwise prompting
approaches have been proposed for LLM-based zero-shot ranking. Our study begins
by thoroughly evaluating these existing approaches within a consistent
experimental framework, considering factors like model size, token consumption,
latency, among others. This first-of-its-kind comparative evaluation of these
approaches allows us to identify the trade-offs between effectiveness and
efficiency inherent in each approach. We find that while Pointwise approaches
score high on efficiency, they suffer from poor effectiveness. Conversely,
Pairwise approaches demonstrate superior effectiveness but incur high
computational overhead. To further enhance the efficiency of LLM-based
zero-shot ranking, we propose a novel Setwise prompting approach. Our approach
reduces the number of LLM inferences and the amount of prompt token consumption
during the ranking procedure, significantly improving the efficiency of
LLM-based zero-shot ranking. We test our method using the TREC DL datasets and
the BEIR zero-shot document ranking benchmark. The empirical results indicate
that our approach considerably reduces computational costs while also retaining
high zero-shot ranking effectiveness.

大型语言模型在零样本文档排序任务中展现出令人印象深刻的效果，本研究通过在一致的实验框架下全面评估 Pointwise、Pairwise 和 Setwise 提示方法，考虑模型大小、令牌消耗、延迟等因素，找出各方法之间存在的效果和效率之间的权衡。我们发现，尽管 Pointwise 方法在效率上得分较高，但效果较差；相反，Pairwise 方法在效果上表现出色，但计算开销较大。为了进一步提高基于大型语言模型的零样本排序的效率，我们提出了一种新颖的 Setwise 提示方法，通过减少排名过程中的 LLM 推理次数和提示令牌的消耗量，显著提高了基于大型语言模型的零样本排序的效率。我们使用 TREC DL 数据集和 BEIR 零样本文档排序基准测试了我们的方法，实证结果表明我们的方法在大幅减少计算成本的同时，同时保持了较高的零样本排序效果。