Contemporary vision benchmarks predominantly consider tasks on which humans
can achieve near-perfect performance. However, humans are frequently presented
with visual data that they cannot classify with 100% certainty, and models
trained on standard vision benchmarks achieve low performance when evaluated on
this data. To address this issue, we introduce a procedure for creating
datasets of ambiguous images and use it to produce SQUID-E ("Squidy"), a
collection of noisy images extracted from videos. All images are annotated with
ground truth values and a test set is annotated with human uncertainty
judgments. We use this dataset to characterize human uncertainty in vision
tasks and evaluate existing visual event classification models. Experimental
results suggest that existing vision models are not sufficiently equipped to
provide meaningful outputs for ambiguous images and that datasets of this
nature can be used to assess and improve such models through model training and
direct evaluation of model calibration. These findings motivate large-scale
ambiguous dataset creation and further research focusing on noisy visual data.

本文提出了一种制作模糊图像数据集的方法，并使用此数据集评估现有视觉事件分类模型。实验结果表明，现有的视觉模型无法针对模糊图像提供有意义的输出，因此需要更多类似模糊图像数据集的创造和模型的研究。