We present a framework for ranking images within their class based on the
strength of spurious cues present. By measuring the gap in accuracy on the
highest and lowest ranked images (we call this spurious gap), we assess
spurious feature reliance for $89$ diverse ImageNet models, finding that even
the best models underperform in images with weak spurious presence. However,
the effect of spurious cues varies far more dramatically across classes,
emphasizing the crucial, often overlooked, class-dependence of the spurious
correlation problem. While most spurious features we observe are clarifying
(i.e. improving test-time accuracy when present, as is typically expected), we
surprisingly find many cases of confusing spurious features, where models
perform better when they are absent. We then close the spurious gap by training
new classification heads on lowly ranked (i.e. without common spurious cues)
images, resulting in improved effective robustness to distribution shifts
(ObjectNet, ImageNet-R, ImageNet-Sketch). We also propose a second metric to
assess feature reliability, finding that spurious features are generally less
reliable than non-spurious (core) ones, though again, spurious features can be
more reliable for certain classes. To enable our analysis, we annotated $5,000$
feature-class dependencies over {\it all} of ImageNet as core or spurious using
minimal human supervision. Finally, we show the feature discovery and
spuriosity ranking framework can be extended to other datasets like CelebA and
WaterBirds in a lightweight fashion with only linear layer training, leading to
discovering a previously unknown racial bias in the Celeb-A hair
classification.

提出了一种框架，用于基于存在的虚假线索以及图像排名，评估 ImageNet 模型的错误关联问题，通过对错误排名和正确排名的图像的准确度差距的测量（称为虚假差距），对 89 个 ImageNet 模型进行评估，发现即使是最好的模型在具有虚假线索存在的图像中也表现不佳，并且这种虚假线索的影响在不同的类别之间差异更大。