In theoretical neuroscience, recent work leverages deep learning tools to
explore how some network attributes critically influence its learning dynamics.
Notably, initial weight distributions with small (resp. large) variance may
yield a rich (resp. lazy) regime, where significant (resp. minor) changes to
network states and representation are observed over the course of learning.
However, in biology, neural circuit connectivity generally has a low-rank
structure and therefore differs markedly from the random initializations
generally used for these studies. As such, here we investigate how the
structure of the initial weights, in particular their effective rank,
influences the network learning regime. Through both empirical and theoretical
analyses, we discover that high-rank initializations typically yield smaller
network changes indicative of lazier learning, a finding we also confirm with
experimentally-driven initial connectivity in recurrent neural networks.
Conversely, low-rank initialization biases learning towards richer learning.
Importantly, however, as an exception to this rule, we find lazier learning can
still occur with a low-rank initialization that aligns with task and data
statistics. Our research highlights the pivotal role of initial weight
structures in shaping learning regimes, with implications for metabolic costs
of plasticity and risks of catastrophic forgetting.

通过理论和实证分析，我们研究发现，初始权重的结构对网络学习机制起着重要影响，具有高秩的初始权重表现出更懒惰的学习，低秩的初始权重倾向于更丰富的学习，同时我们还发现与任务和数据统计信息一致的低秩初始权重仍然可能导致懒散学习。这一研究突出了初始权重结构在塑性代谢成本和灾难性遗忘风险方面的关键作用。