Despite large language models' (LLMs) recent advancements, their bias and hallucination issues persist, and their ability to offer consistent preferential rankings remains underexplored. This study investigates the capacity of LLMs to provide consistent ordinal preferences, a crucial aspect in scenarios with dense decision space or lacking absolute answers. We introduce a formalization of consistency based on order theory, outlining criteria such as transitivity, asymmetry, reversibility, and independence from irrelevant alternatives. Our diagnostic experiments on selected state-of-the-art LLMs reveal their inability to meet these criteria, indicating a strong positional bias and poor transitivity, with preferences easily swayed by irrelevant alternatives. These findings highlight a significant inconsistency in LLM-generated preferential rankings, underscoring the need for further research to address these limitations.

本研究解决了大型语言模型在提供一致的偏好排序方面的不足，尤其是在缺乏绝对答案的情况下。通过基于序理论的形式化，我们验证了当前先进的语言模型在满足一致性标准方面的能力，结果显示这些模型存在显著的不一致，提示需要进一步的研究以克服这些局限性。

大型语言模型偏好排序不一致性的测量