The training dynamics of linear networks are well studied in two distinct
setups: the lazy regime and balanced/active regime, depending on the
initialization and width of the network. We provide a surprisingly simple
unyfing formula for the evolution of the learned matrix that contains as
special cases both lazy and balanced regimes but also a mixed regime in between
the two. In the mixed regime, a part of the network is lazy while the other is
balanced. More precisely the network is lazy along singular values that are
below a certain threshold and balanced along those that are above the same
threshold. At initialization, all singular values are lazy, allowing for the
network to align itself with the task, so that later in time, when some of the
singular value cross the threshold and become active they will converge rapidly
(convergence in the balanced regime is notoriously difficult in the absence of
alignment). The mixed regime is the `best of both worlds': it converges from
any random initialization (in contrast to balanced dynamics which require
special initialization), and has a low rank bias (absent in the lazy dynamics).
This allows us to prove an almost complete phase diagram of training behavior
as a function of the variance at initialization and the width, for a MSE
training task.

线性网络的训练动力学在两个不同的设置中得到了深入研究：懒惰模式和均衡 / 活跃模式，具体取决于网络的初始化和宽度。我们提供了一个令人惊讶地简单的统一公式，用于描述学习矩阵的演化，其中包括懒惰模式和均衡模式，还有两者之间的混合模式。

线性网络中的混合动力学：统一惰性和活跃状态

Mixed Dynamics In Linear Networks: Unifying the Lazy and Active Regimes

In these six lectures, we examine what can be learnt about the behavior of
multi-layer neural networks from the analysis of linear models. We first recall
the correspondence between neural networks and linear models via the so-called
lazy regime. We then review four models for linearized neural networks: linear
regression with concentrated features, kernel ridge regression, random feature
model and neural tangent model. Finally, we highlight the limitations of the
linear theory and discuss how other approaches can overcome them.

从线性模型分析多层神经网络的行为，并回顾其与线性模型之间的对应关系，以及线性化神经网络的四种模型：线性回归、核岭回归、随机特征模型和神经切线模型，以及线性理论的局限性和其他方法如何克服这些局限性。