We study Leaky ResNets, which interpolate between ResNets ($\tilde{L}=0$) and
Fully-Connected nets ($\tilde{L}\to\infty$) depending on an 'effective depth'
hyper-parameter $\tilde{L}$. In the infinite depth limit, we study
'representation geodesics' $A_{p}$: continuous paths in representation space
(similar to NeuralODEs) from input $p=0$ to output $p=1$ that minimize the
parameter norm of the network. We give a Lagrangian and Hamiltonian
reformulation, which highlight the importance of two terms: a kinetic energy
which favors small layer derivatives $\partial_{p}A_{p}$ and a potential energy
that favors low-dimensional representations, as measured by the 'Cost of
Identity'. The balance between these two forces offers an intuitive
understanding of feature learning in ResNets. We leverage this intuition to
explain the emergence of a bottleneck structure, as observed in previous work:
for large $\tilde{L}$ the potential energy dominates and leads to a separation
of timescales, where the representation jumps rapidly from the high dimensional
inputs to a low-dimensional representation, move slowly inside the space of
low-dimensional representations, before jumping back to the potentially
high-dimensional outputs. Inspired by this phenomenon, we train with an
adaptive layer step-size to adapt to the separation of timescales.

我们研究了 Leaky ResNets，它在 ResNets（$\tilde {L}=0$）和全连接网络（$\tilde {L} \to \infty$）之间进行内插，具体取决于一个 “有效深度” 超参数 $\tilde {L}$。在无限深度极限下，我们研究了表示空间中的连续路径 $A_{p}$（类似于 NeuralODEs），这些路径从输入 $p=0$ 到输出 $p=1$，最小化网络的参数范数。我们给出了一个拉格朗日和哈密顿重述，突出了两个重要因素：一个动能，它偏爱小层导数 $\partial_{p} A_{p}$，一个势能，它偏爱低维表示，用 “恒等成本” 进行测量。这两种力之间的平衡提供了对 ResNets 中特征学习的直观理解。我们利用这种直观理解来解释之前工作中观察到的瓶颈结构的出现：对于大的 $\tilde {L}$，势能占主导地位，导致时间尺度的分离，表示空间从高维输入迅速跳跃到低维表示，然后在低维表示空间内缓慢移动，最后又跳回可能是高维的输出。受到这一现象的启发，我们使用自适应层步长进行训练，以适应时间尺度的分离。

特征学习的哈密顿力学：漏洞结构在渗漏 ResNets 中的应用

Hamiltonian Mechanics of Feature Learning: Bottleneck Structure in Leaky  ResNets

We show at a physics level of rigor that Bayesian inference with a fully
connected neural network and a shaped nonlinearity of the form $\phi(t) = t +
\psi t^3/L$ is (perturbatively) solvable in the regime where the number of
training datapoints $P$ , the input dimension $N_0$, the network layer widths
$N$, and the network depth $L$ are simultaneously large. Our results hold with
weak assumptions on the data; the main constraint is that $P < N_0$. We provide
techniques to compute the model evidence and posterior to arbitrary order in
$1/N$ and at arbitrary temperature. We report the following results from the
first-order computation:
1. When the width $N$ is much larger than the depth $L$ and training set size
$P$, neural network Bayesian inference coincides with Bayesian inference using
a kernel. The value of $\psi$ determines the curvature of a sphere, hyperbola,
or plane into which the training data is implicitly embedded under the feature
map.
2. When $LP/N$ is a small constant, neural network Bayesian inference departs
from the kernel regime. At zero temperature, neural network Bayesian inference
is equivalent to Bayesian inference using a data-dependent kernel, and $LP/N$
serves as an effective depth that controls the extent of feature learning.
3. In the restricted case of deep linear networks ($\psi=0$) and noisy data,
we show a simple data model for which evidence and generalization error are
optimal at zero temperature. As $LP/N$ increases, both evidence and
generalization further improve, demonstrating the benefit of depth in benign
overfitting.

在大量训练数据、输入维度、网络层宽度和网络深度同时很大的情况下，我们展示了贝叶斯推断与全连接神经网络和形状非线性的关系，并提供了计算模型证据和后验的技术，结果表明神经网络贝叶斯推断与使用核函数的贝叶斯推断相一致，当网络层宽度大于深度和训练集大小时，神经网络贝叶斯推断的深度是一个有效的参数。