We study Leaky ResNets, which interpolate between ResNets ($\tilde{L}=0$) and
Fully-Connected nets ($\tilde{L}\to\infty$) depending on an 'effective depth'
hyper-parameter $\tilde{L}$. In the infinite depth limit, we study
'representation geodesics' $A_{p}$: continuous paths in representation space
(similar to NeuralODEs) from input $p=0$ to output $p=1$ that minimize the
parameter norm of the network. We give a Lagrangian and Hamiltonian
reformulation, which highlight the importance of two terms: a kinetic energy
which favors small layer derivatives $\partial_{p}A_{p}$ and a potential energy
that favors low-dimensional representations, as measured by the 'Cost of
Identity'. The balance between these two forces offers an intuitive
understanding of feature learning in ResNets. We leverage this intuition to
explain the emergence of a bottleneck structure, as observed in previous work:
for large $\tilde{L}$ the potential energy dominates and leads to a separation
of timescales, where the representation jumps rapidly from the high dimensional
inputs to a low-dimensional representation, move slowly inside the space of
low-dimensional representations, before jumping back to the potentially
high-dimensional outputs. Inspired by this phenomenon, we train with an
adaptive layer step-size to adapt to the separation of timescales.

我们研究了 Leaky ResNets，它在 ResNets（$\tilde {L}=0$）和全连接网络（$\tilde {L} \to \infty$）之间进行内插，具体取决于一个 “有效深度” 超参数 $\tilde {L}$。在无限深度极限下，我们研究了表示空间中的连续路径 $A_{p}$（类似于 NeuralODEs），这些路径从输入 $p=0$ 到输出 $p=1$，最小化网络的参数范数。我们给出了一个拉格朗日和哈密顿重述，突出了两个重要因素：一个动能，它偏爱小层导数 $\partial_{p} A_{p}$，一个势能，它偏爱低维表示，用 “恒等成本” 进行测量。这两种力之间的平衡提供了对 ResNets 中特征学习的直观理解。我们利用这种直观理解来解释之前工作中观察到的瓶颈结构的出现：对于大的 $\tilde {L}$，势能占主导地位，导致时间尺度的分离，表示空间从高维输入迅速跳跃到低维表示，然后在低维表示空间内缓慢移动，最后又跳回可能是高维的输出。受到这一现象的启发，我们使用自适应层步长进行训练，以适应时间尺度的分离。

特征学习的哈密顿力学：漏洞结构在渗漏 ResNets 中的应用

Hamiltonian Mechanics of Feature Learning: Bottleneck Structure in Leaky  ResNets

Previous work has shown that DNNs with large depth $L$ and
$L_{2}$-regularization are biased towards learning low-dimensional
representations of the inputs, which can be interpreted as minimizing a notion
of rank $R^{(0)}(f)$ of the learned function $f$, conjectured to be the
Bottleneck rank. We compute finite depth corrections to this result, revealing
a measure $R^{(1)}$ of regularity which bounds the pseudo-determinant of the
Jacobian $\left|Jf(x)\right|_{+}$ and is subadditive under composition and
addition. This formalizes a balance between learning low-dimensional
representations and minimizing complexity/irregularity in the feature maps,
allowing the network to learn the `right' inner dimension. We also show how
large learning rates also control the regularity of the learned function.
Finally, we use these theoretical tools to prove the conjectured bottleneck
structure in the learned features as $L\to\infty$: for large depths, almost all
hidden representations concentrates around $R^{(0)}(f)$-dimensional
representations. These limiting low-dimensional representation can be described
using the second correction $R^{(2)}$.

该研究旨在证明具有大深度和 L2 正则化的 DNN 在学习输入时会产生 “瓶颈结构”（低维表示），并介绍了一种衡量网络内在维度和复杂性 / 不规则性之间平衡的方法。