Large language models (LLMs) exhibit positional bias in how they use context,
which especially complicates listwise ranking. To address this, we propose
permutation self-consistency, a form of self-consistency over ranking list
outputs of black-box LLMs. Our key idea is to marginalize out different list
orders in the prompt to produce an order-independent ranking with less
positional bias. First, given some input prompt, we repeatedly shuffle the list
in the prompt and pass it through the LLM while holding the instructions the
same. Next, we aggregate the resulting sample of rankings by computing the
central ranking closest in distance to all of them, marginalizing out prompt
order biases in the process. Theoretically, we prove the robustness of our
method, showing convergence to the true ranking in the presence of random
perturbations. Empirically, on five list-ranking datasets in sorting and
passage reranking, our approach improves scores from conventional inference by
up to 7-18% for GPT-3.5 and 8-16% for LLaMA v2 (70B), surpassing the previous
state of the art in passage reranking. Our code is at
this https URL

大型语言模型 (LLMs) 在使用上下文方面存在位置偏倚，尤其是在列表排序方面。为解决这个问题，我们提出了置换自一致性，这是一种对黑盒 LLMs 的排序列表输出的自一致性形式。我们的关键思想是通过将提示中的不同列表顺序边际化来产生一个无序偏倚更少的排序。我们在排序和段落重新排序的五个列表排序数据集上进行了实证实验，我们的方法在 GPT-3.5 上相比传统推理提高了 7-18%，在 LLaMA v2 (70B) 上提高了 8-16%，超过了先前的段落重新排序最先进技术。