Convolutional neural networks (CNNs) are one of the most successful computer
vision systems to solve object recognition. Furthermore, CNNs have major
applications in understanding the nature of visual representations in the human
brain. Yet it remains poorly understood how CNNs actually make their decisions,
what the nature of their internal representations is, and how their recognition
strategies differ from humans. Specifically, there is a major debate about the
question of whether CNNs primarily rely on surface regularities of objects, or
whether they are capable of exploiting the spatial arrangement of features,
similar to humans. Here, we develop a novel feature-scrambling approach to
explicitly test whether CNNs use the spatial arrangement of features (i.e.
object parts) to classify objects. We combine this approach with a systematic
manipulation of effective receptive field sizes of CNNs as well as minimal
recognizable configurations (MIRCs) analysis. In contrast to much previous
literature, we provide evidence that CNNs are in fact capable of using
relatively long-range spatial relationships for object classification.
Moreover, the extent to which CNNs use spatial relationships depends heavily on
the dataset, e.g. texture vs. sketch. In fact, CNNs even use different
strategies for different classes within heterogeneous datasets (ImageNet),
suggesting CNNs have a continuous spectrum of classification strategies.
Finally, we show that CNNs learn the spatial arrangement of features only up to
an intermediate level of granularity, which suggests that intermediate rather
than global shape features provide the optimal trade-off between sensitivity
and specificity in object classification. These results provide novel insights
into the nature of CNN representations and the extent to which they rely on the
spatial arrangement of features for object classification.

该研究使用特征打乱的方法，结合 CNNs 的最小可识别构型分析和有效接受域尺寸的系统控制，探索了 CNNs 在对象分类中是否使用特征的空间排列，并且发现它们实际上能够利用相对较远的空间关系进行对象分类，同时证明 CNNs 使用的空间关系程度非常依赖于特定数据集和该数据集中不同类别对象的分类策略也不同。最后，研究还发现 CNNs 只能学习特征的中间粒度的空间排列，这提示了在对象分类中中间层次的形状特征提供了敏感度和特异性之间的最佳平衡点。