Deep neural networks have been demonstrated to achieve phenomenal success in
many domains, and yet their inner mechanisms are not well understood. In this
paper, we investigate the curvature of image manifolds, i.e., the manifold
deviation from being flat in its principal directions. We find that
state-of-the-art trained convolutional neural networks for image classification
have a characteristic curvature profile along layers: an initial steep
increase, followed by a long phase of a plateau, and followed by another
increase. In contrast, this behavior does not appear in untrained networks in
which the curvature flattens. We also show that the curvature gap between the
last two layers has a strong correlation with the generalization capability of
the network. Moreover, we find that the intrinsic dimension of latent codes is
not necessarily indicative of curvature. Finally, we observe that common
regularization methods such as mixup yield flatter representations when
compared to other methods. Our experiments show consistent results over a
variety of deep learning architectures and multiple data sets. Our code is
publicly available at this https URL

本文探究了图像流形的曲率，发现卷积神经网络在分类任务中的整体曲率表现具有特殊的特点，即初始速度的加速，接着进入了一个较长的平稳期和之后的再加速。此外，我们发现最后两层图层的曲率缺口与网络的泛化能力有很强的相关性，并观察到常见的正则化方法，如 mixup，在相比其他方法时，呈现出更平的表示特征。