We leverage probabilistic models of neural representations to investigate how residual networks fit classes. To this end, we estimate class-conditional density models for representations learned by deep ResNets. We then use these models to characterize distributions of representations across learned classes. Surprisingly, we find that classes in the investigated models are not fitted in an uniform way. On the contrary: we uncover two groups of classes that are fitted with markedly different distributions of representations. These distinct modes of class-fitting are evident only in the deeper layers of the investigated models, indicating that they are not related to low-level image features. We show that the uncovered structure in neural representations correlate with memorization of training examples and adversarial robustness. Finally, we compare class-conditional distributions of neural representations between memorized and typical examples. This allows us to uncover where in the network structure class labels arise for memorized and standard inputs.

本文探讨了使用概率模型分析神经表示的方法，揭示了深度残差网络对类别的拟合存在两种不同的分布模式，这种结构与训练样例的记忆化和对抗性鲁棒性的相关性有关，并通过比较神经表示的类别条件分布，揭示了类标签在网络结构中的出现位置。

残差卷积神经网络中的分类拟合模式揭示