The two-alternative forced choice (2AFC) experimental setup is popular in the
visual perception literature, where practitioners aim to understand how human
observers perceive distances within triplets that consist of a reference image
and two distorted versions of that image. In the past, this had been conducted
in controlled environments, with a tournament-style algorithm dictating which
images are shown to each participant to rank the distorted images. Recently,
crowd-sourced perceptual datasets have emerged, with no images shared between
triplets, making ranking impossible. Evaluating perceptual distances using this
data is non-trivial, relying on reducing the collection of judgements on a
triplet to a binary decision -- which is suboptimal and prone to misleading
conclusions. Instead, we statistically model the underlying decision-making
process during 2AFC experiments using a binomial distribution. We use maximum
likelihood estimation to fit a distribution to the perceptual judgements,
conditioned on the perceptual distance to test and impose consistency and
smoothness between our empirical estimates of the density. This way, we can
evaluate a different number of judgements per triplet, and can calculate
metrics such as likelihoods of judgements according to a set of distances --
key ingredients that neural network counterparts lack.

本文使用双选强迫选择实验（2AFC）的统计建模方法，通过最大似然估计来适配分布，对感知判断进行评估，并计算不同距离下的判断可能性，从而解决了在感知数据中减少判断数目的问题，同时强调了神经网络方法所缺少的关键要素。

通过拟合二项分布评估感知距离的两个选择数据

Evaluating Perceptual Distances by Fitting Binomial Distributions to  Two-Alternative Forced Choice Data

While abundant research has been conducted on improving high-level visual
understanding and reasoning capabilities of large multimodal models~(LMMs),
their visual quality assessment~(IQA) ability has been relatively
under-explored. Here we take initial steps towards this goal by employing the
two-alternative forced choice~(2AFC) prompting, as 2AFC is widely regarded as
the most reliable way of collecting human opinions of visual quality.
Subsequently, the global quality score of each image estimated by a particular
LMM can be efficiently aggregated using the maximum a posterior estimation.
Meanwhile, we introduce three evaluation criteria: consistency, accuracy, and
correlation, to provide comprehensive quantifications and deeper insights into
the IQA capability of five LMMs. Extensive experiments show that existing LMMs
exhibit remarkable IQA ability on coarse-grained quality comparison, but there
is room for improvement on fine-grained quality discrimination. The proposed
dataset sheds light on the future development of IQA models based on LMMs. The
codes will be made publicly available at this https URL

通过使用 2AFC 提示，评估了大型多模态模型（LMMs）的图像质量评估（IQA）能力，并引入了三个评估标准，结果显示现有的 LMMs 在粗粒度的质量比较上表现出较高的 IQA 能力，但在细粒度的质量判别方面仍有提升空间。