Biological and artificial neural networks develop internal representations that enable them to perform complex tasks. In artificial networks, the effectiveness of these models relies on their ability to build task specific representation, a process influenced by interactions among datasets, architectures, initialization strategies, and optimization algorithms. Prior studies highlight that different initializations can place networks in either a lazy regime, where representations remain static, or a rich/feature learning regime, where representations evolve dynamically. Here, we examine how initialization influences learning dynamics in deep linear neural networks, deriving exact solutions for lambda-balanced initializations-defined by the relative scale of weights across layers. These solutions capture the evolution of representations and the Neural Tangent Kernel across the spectrum from the rich to the lazy regimes. Our findings deepen the theoretical understanding of the impact of weight initialization on learning regimes, with implications for continual learning, reversal learning, and transfer learning, relevant to both neuroscience and practical applications.

本文研究了初始化对深线性神经网络学习动态的影响，填补了关于权重初始化如何影响网络表示演变的知识空白。作者提出了一种新的方法，通过拉姆达平衡初始化精确推导出学习动态的解决方案，揭示了从懒惰到富有学习模式的转变。研究结果加深了对初始化对学习模式影响的理论理解，具有持续学习、逆转学习和迁移学习的广泛应用潜力。

从懒惰到富有：深线性网络中的精确学习动态