Ranking documents using Large Language Models (LLMs) by directly feeding the
query and candidate documents into the prompt is an interesting and practical
problem. However, there has been limited success so far, as researchers have
found it difficult to outperform fine-tuned baseline rankers on benchmark
datasets. We analyze pointwise and listwise ranking prompts used by existing
methods and argue that off-the-shelf LLMs do not fully understand these ranking
formulations, possibly due to the nature of how LLMs are trained. In this
paper, we propose to significantly reduce the burden on LLMs by using a new
technique called Pairwise Ranking Prompting (PRP). Our results are the first in
the literature to achieve state-of-the-art ranking performance on standard
benchmarks using moderate-sized open-sourced LLMs. On TREC-DL2020, PRP based on
the Flan-UL2 model with 20B parameters outperforms the previous best approach
in the literature, which is based on the blackbox commercial GPT-4 that has 50x
(estimated) model size, by over 5% at NDCG@1. On TREC-DL2019, PRP is only
inferior to the GPT-4 solution on the NDCG@5 and NDCG@10 metrics, while
outperforming other existing solutions, such as InstructGPT which has 175B
parameters, by over 10% for nearly all ranking metrics. Furthermore, we propose
several variants of PRP to improve efficiency and show that it is possible to
achieve competitive results even with linear complexity. We also discuss other
benefits of PRP, such as supporting both generation and scoring LLM APIs, as
well as being insensitive to input ordering.

本文提出了一种新的技术叫做 Pairwise Ranking Prompting (PR)，通过使用适当的排名提示，减少了 LLMs 的负担，使用 moderate-sized open-sourced LLMs 在标准的基准测试中实现了最先进的排名表现。