We study deep neural networks (DNNs) trained on natural image data with entirely random labels. Despite its popularity in the literature, where it is often used to study memorization, generalization, and other phenomena, little is known about what DNNs learn in this setting. In this paper, we show analytically for convolutional and fully connected networks that an alignment between the principal components of network parameters and data takes place when training with random labels. We study this alignment effect by investigating neural networks pre-trained on randomly labelled image data and subsequently fine-tuned on disjoint datasets with random or real labels. We show how this alignment produces a positive transfer: networks pre-trained with random labels train faster downstream compared to training from scratch even after accounting for simple effects, such as weight scaling. We analyze how competing effects, such as specialization at later layers, may hide the positive transfer. These effects are studied in several network architectures, including VGG16 and ResNet18, on CIFAR10 and ImageNet.

本文研究了使用纯随机标签培训的自然图像数据的深度神经网络 (DNNs)，通过分析卷积和全连接网络的主成分与数据的对齐作用，论证了在此设置中发生了什么。研究人员还研究了预先经过随机标记的图像数据进行预训练的神经网络以及随机和真实标签上进行微调的神经网络，发现此对准过程产生了正向迁移，特别是在CIFAR10和ImageNet上的VGG16和ResNet18等多种网络体系结构中。

当随机标签训练神经网络时，神经网络学习到了什么？