The ResNet architecture has been widely adopted in deep learning due to its
significant boost to performance through the use of simple skip connections,
yet the underlying mechanisms leading to its success remain largely unknown. In
this paper, we conduct a thorough empirical study of the ResNet architecture in
classification tasks by linearizing its constituent residual blocks using
Residual Jacobians and measuring their singular value decompositions. Our
measurements reveal a process called Residual Alignment (RA) characterized by
four properties:
(RA1) intermediate representations of a given input are equispaced on a line,
embedded in high dimensional space, as observed by Gai and Zhang [2021];
(RA2) top left and right singular vectors of Residual Jacobians align with
each other and across different depths;
(RA3) Residual Jacobians are at most rank C for fully-connected ResNets,
where C is the number of classes; and
(RA4) top singular values of Residual Jacobians scale inversely with depth.
RA consistently occurs in models that generalize well, in both
fully-connected and convolutional architectures, across various depths and
widths, for varying numbers of classes, on all tested benchmark datasets, but
ceases to occur once the skip connections are removed. It also provably occurs
in a novel mathematical model we propose. This phenomenon reveals a strong
alignment between residual branches of a ResNet (RA2+4), imparting a highly
rigid geometric structure to the intermediate representations as they progress
linearly through the network (RA1) up to the final layer, where they undergo
Neural Collapse.

ResNet 架构在深度学习中得到广泛采用，通过使用简单的跳跃连接显著提高性能，但其成功背后的机制仍然大部分未知。本文通过使用剩余雅可比矩阵对其构成的残差块进行线性化，并测量其奇异值分解，在分类任务中进行了对 ResNet 架构的彻底实证研究。我们的测量结果揭示了一个被称为 Residual Alignment (RA) 的过程，该过程具有四个特征，包括中间表示在高维空间中等间隔地嵌入在一条直线上（RA1），残差雅可比左右奇异向量与各个深度之间以及不同深度之间对齐（RA2），全连接 ResNets 的残差雅可比最多具有 C 阶可逆性，其中 C 为类别数量（RA3），残差雅可比的前奇异值与深度成反比（RA4）。RA 在那些泛化良好的模型中始终存在，无论是全连接还是卷积架构，无论深度和宽度如何，无论类别数量如何，而且适用于所有经过测试的基准数据集，但一旦跳跃连接被移除，RA 将不再出现。我们还提出了一个新的数学模型，该现象在我们提出的数学模型中得到验证。这种现象揭示了 ResNet 的残差分支之间的强大对齐性（RA2+4），将中间表示沿着网络线性推进（RA1）直到最后一层，在最后一层中，它们经历了神经崩溃（Neural Collapse）。