Large Language Models (LLMs) demonstrate impressive effectiveness in zero-shot document ranking tasks. Pointwise, Pairwise, and Listwise prompting approaches have been proposed for LLM-based zero-shot ranking. Our study begins by thoroughly evaluating these existing approaches within a consistent experimental framework, considering factors like model size, token consumption, latency, among others. This first-of-its-kind comparative evaluation of these approaches allows us to identify the trade-offs between effectiveness and efficiency inherent in each approach. We find that while Pointwise approaches score high on efficiency, they suffer from poor effectiveness. Conversely, Pairwise approaches demonstrate superior effectiveness but incur high computational overhead. To further enhance the efficiency of LLM-based zero-shot ranking, we propose a novel Setwise prompting approach. Our approach reduces the number of LLM inferences and the amount of prompt token consumption during the ranking procedure, significantly improving the efficiency of LLM-based zero-shot ranking. We test our method using the TREC DL datasets and the BEIR zero-shot document ranking benchmark. The empirical results indicate that our approach considerably reduces computational costs while also retaining high zero-shot ranking effectiveness.

大型语言模型在零样本文档排序任务中展现出令人印象深刻的效果，本研究通过在一致的实验框架下全面评估Pointwise、Pairwise和Setwise提示方法，考虑模型大小、令牌消耗、延迟等因素，找出各方法之间存在的效果和效率之间的权衡。我们发现，尽管Pointwise方法在效率上得分较高，但效果较差；相反，Pairwise方法在效果上表现出色，但计算开销较大。为了进一步提高基于大型语言模型的零样本排序的效率，我们提出了一种新颖的Setwise提示方法，通过减少排名过程中的LLM推理次数和提示令牌的消耗量，显著提高了基于大型语言模型的零样本排序的效率。我们使用TREC DL数据集和BEIR零样本文档排序基准测试了我们的方法，实证结果表明我们的方法在大幅减少计算成本的同时，同时保持了较高的零样本排序效果。

一种用于大语言模型的高效零-shot排名的集合方法