Ranking passages by prompting a large language model (LLM) can achieve
promising performance in modern information retrieval (IR) systems. A common
approach is to sort the ranking list by prompting LLMs for pairwise comparison.
However, sorting-based methods require consistent comparisons to correctly sort
the passages, which we show that LLMs often violate. We identify two kinds of
intrinsic inconsistency in LLM-based pairwise comparisons: order inconsistency
which leads to conflicting results when switching the passage order, and
transitive inconsistency which leads to non-transitive triads among all
preference pairs. In this paper, we propose LLM-RankFusion, an LLM-based
ranking framework that mitigates these inconsistencies and produces a robust
ranking list. LLM-RankFusion mitigates order inconsistency using in-context
learning (ICL) to demonstrate order-agnostic comparisons and calibration to
estimate the underlying preference probability between two passages. We then
address transitive inconsistency by aggregating the ranking results from
multiple rankers. In our experiments, we empirically show that LLM-RankFusion
can significantly reduce inconsistent pairwise comparison results, and improve
the ranking quality by making the final ranking list more robust.

使用语言模型比较并排序排名的矛盾问题，我们提出了一种使用 LLM-RankFusion 的方法，它有效地减少了排序的不一致性，提高了排名质量。