Recent studies have shown that many important aspects of neural network
learning take place within the very earliest iterations or epochs of training.
For example, sparse, trainable sub-networks emerge (Frankle et al., 2019),
gradient descent moves into a small subspace (Gur-Ari et al., 2018), and the
network undergoes a critical period (Achille et al., 2019). Here, we examine
the changes that deep neural networks undergo during this early phase of
training. We perform extensive measurements of the network state during these
early iterations of training and leverage the framework of Frankle et al.
(2019) to quantitatively probe the weight distribution and its reliance on
various aspects of the dataset. We find that, within this framework, deep
networks are not robust to reinitializing with random weights while maintaining
signs, and that weight distributions are highly non-independent even after only
a few hundred iterations. Despite this behavior, pre-training with blurred
inputs or an auxiliary self-supervised task can approximate the changes in
supervised networks, suggesting that these changes are not inherently
label-dependent, though labels significantly accelerate this process. Together,
these results help to elucidate the network changes occurring during this
pivotal initial period of learning.

本文研究神经网络学习的早期阶段，分析了神经网络在此期间的变化，发现深度网络在使用随机权重重初始化时不具有稳健性，但利用模糊输入或辅助自监督任务进行预训练即可近似监督网络的变化。