Kahneman & Tversky's $\textit{prospect theory}$ tells us that humans perceive random variables in a biased but well-defined manner; for example, humans are famously loss-averse. We show that objectives for aligning LLMs with human feedback implicitly incorporate many of these biases -- the success of these objectives (e.g., DPO) over cross-entropy minimization can partly be ascribed to them being $\textit{human-aware loss functions}$ (HALOs). However, the utility functions these methods attribute to humans still differ from those in the prospect theory literature. Using a Kahneman-Tversky model of human utility, we propose a HALO that directly maximizes the utility of generations instead of maximizing the log-likelihood of preferences, as current methods do. We call this approach Kahneman-Tversky Optimization (KTO), and it matches or exceeds the performance of preference-based methods at scales from 1B to 30B. Crucially, KTO does not need preferences -- only a binary signal of whether an output is desirable or undesirable for a given input. This makes it far easier to use in the real world, where preference data is scarce and expensive.

人们的感知有偏见，Kahneman-Tversky前景理论告诉我们，对齐LLMs和人类反馈的目标隐含地包含了许多这些偏见 - 这些目标的成功部分可以归因于它们是人类感知感知的损失函数（HALOs）；我们使用Kahneman-Tversky模型提出了一种HALO方法，该方法直接最大化生成物的效用而不是最大化需求偏好的对数似然，这种方法叫做Kahneman-Tversky优化（KTO），在从1B到30B的规模上与基于偏好的方法的性能相匹配或超过；关键是，KTO不需要偏好 - 只需要对于给定输入的输出是可取还是不可取的二进制信号，这使得在偏好数据稀缺且昂贵的现实世界中更容易使用。

KTO：模型对齐作为前景理论优化