A key factor in the success of deep neural networks is the ability to scale models to improve performance by varying the architecture depth and width. This simple property of neural network design has resulted in highly effective architectures for a variety of tasks. Nevertheless, there is limited understanding of effects of depth and width on the learned representations. In this paper, we study this fundamental question. We begin by investigating how varying depth and width affects model hidden representations, finding a characteristic block structure in the hidden representations of larger capacity (wider or deeper) models. We demonstrate that this block structure arises when model capacity is large relative to the size of the training set, and is indicative of the underlying layers preserving and propagating the dominant principal component of their representations. This discovery has important ramifications for features learned by different models, namely, representations outside the block structure are often similar across architectures with varying widths and depths, but the block structure is unique to each model. We analyze the output predictions of different model architectures, finding that even when the overall accuracy is similar, wide and deep models exhibit distinctive error patterns and variations across classes.

本篇研究通过探究深度和宽度对模型隐藏表示的影响，发现一个特征块结构，这个结构是相对于训练集大小而言容量较大（更宽/更深）的模型中隐藏表示的一种保留和传播主要成分的方式。该发现对不同模型学习的特征有重要影响，其代表性的块结构对于每个模型来说是独一无二的。最后，我们分析了不同模型体系结构的输出预测，发现即使整体准确率相似，宽而深的模型也表现出不同类别之间鲜明的错误模式和差异。

广而深的神经网络是否学习相同的内容？揭示不同宽度和深度下神经网络表示的变化