We introduce a benchmark to directly evaluate the alignment between human observers and vision models on a 3D shape inference task. We leverage an experimental design from the cognitive sciences which requires zero-shot visual inferences about object shape: given a set of images, participants identify which contain the same/different objects, despite considerable viewpoint variation. We draw from a diverse range of images that include common objects (e.g., chairs) as well as abstract shapes (i.e., procedurally generated `nonsense' objects). After constructing over 2000 unique image sets, we administer these tasks to human participants, collecting 35K trials of behavioral data from over 500 participants. This includes explicit choice behaviors as well as intermediate measures, such as reaction time and gaze data. We then evaluate the performance of common vision models (e.g., DINOv2, MAE, CLIP). We find that humans outperform all models by a wide margin. Using a multi-scale evaluation approach, we identify underlying similarities and differences between models and humans: while human-model performance is correlated, humans allocate more time/processing on challenging trials. All images, data, and code can be accessed via our project page.

本研究解决了人类观察者与视觉模型在3D形状推理任务中的对齐问题。通过借鉴认知科学的实验设计，构建了超过2000个独特的图像集，并收集了来自500多名参与者的35K试验行为数据，结果显示人类在此任务中的表现显著优于所有视觉模型。这一发现强调了人类在应对复杂试验时的独特处理能力，潜在影响深远。

评估人类与图像模型中的多视角对象一致性