In this paper, we elucidate how representations in deep neural networks (DNNs) evolve during training. We focus on overparameterized learning settings where the training continues much after the trained DNN starts to perfectly fit its training data. We examine the evolution of learned representations along the entire training process, including its perfect fitting regime, and with respect to the epoch-wise double descent phenomenon. We explore the representational similarity of DNN layers, each layer with respect to its own representations throughout the training process. For this, we use two similarity metrics: (1) The centered kernel alignment (CKA) similarity; (2) Similarity of decision regions of linear classifier probes that we train for the DNN layers. Our extensive experiments discover training dynamics patterns that can emerge in layers depending on the relative layer-depth, DNN width, and architecture. We show that representations at the deeper layers evolve much more in the training when an epoch-wise double descent occurs. For Vision Transformer, we show that the perfect fitting threshold creates a transition in the evolution of representations across all the encoder blocks.

通过揭示深度神经网络（DNNs）在训练过程中的演变，本文重点关注在过度参数化学习环境下，训练持续进行的情况，探讨了学习表示的演变以及层级之间的相似性和线性分类器探针的相似度等指标，并发现了与相对层级深度、DNN宽度和架构有关的训练动态模式。同时，对Vision Transformer进行了研究，展示了完美拟合阈值在编码器块中表示演变方面的转变。

完美拟合对表示学习的影响：深度神经网络中的表示训练动态