Similar to humans and animals, deep artificial neural networks exhibit
critical periods during which a temporary stimulus deficit can impair the
development of a skill. The extent of the impairment depends on the onset and
length of the deficit window, as in animal models, and on the size of the
neural network. Deficits that do not affect low-level statistics, such as
vertical flipping of the images, have no lasting effect on performance and can
be overcome with further training. To better understand this phenomenon, we use
the Fisher Information of the weights to measure the effective connectivity
between layers of a network during training. Counterintuitively, information
rises rapidly in the early phases of training, and then decreases, preventing
redistribution of information resources in a phenomenon we refer to as a loss
of "Information Plasticity". Our analysis suggests that the first few epochs
are critical for the creation of strong connections that are optimal relative
to the input data distribution. Once such strong connections are created, they
do not appear to change during additional training. These findings suggest that
the initial learning transient, under-scrutinized compared to asymptotic
behavior, plays a key role in determining the outcome of the training process.
Our findings, combined with recent theoretical results in the literature, also
suggest that forgetting (decrease of information in the weights) is critical to
achieving invariance and disentanglement in representation learning. Finally,
critical periods are not restricted to biological systems, but can emerge
naturally in learning systems, whether biological or artificial, due to
fundamental constrains arising from learning dynamics and information
processing.

通过研究神经网络的有效连接和信息分布的变化，发现网络的第一阶段训练对于后续模型的性能具有决定性的影响，同时遗忘（权重的信息减少）是达到表征学习中的不变性和解缠的关键，训练过程中出现的缺陷期不仅限于生物系统，也出现在学习动态和信息处理基础上的人工智能系统中。