A robust theoretical framework that can describe and predict the
generalization ability of deep neural networks (DNNs) in general circumstances
remains elusive. Classical attempts have produced complexity metrics that rely
heavily on global measures of compactness and capacity with little
investigation into the effects of sub-component collaboration. We demonstrate
intriguing regularities in the activation patterns of the hidden nodes within
fully-connected feedforward networks. By tracing the origin of these patterns,
we show how such networks can be viewed as the combination of two information
processing systems: one continuous and one discrete. We describe how these two
systems arise naturally from the gradient-based optimization process, and
demonstrate the classification ability of the two systems, individually and in
collaboration. This perspective on DNN classification offers a novel way to
think about generalization, in which different subsets of the training data are
used to train distinct classifiers; those classifiers are then combined to
perform the classification task, and their consistency is crucial for accurate
classification.

本文探讨了全连接前馈神经网络中隐藏节点的激活模式，提出了从两个信息处理系统，即连续性和离散性的角度来看待这种网络分类能力的新方式，并且重点突出了这两者搭配的重要性，为深度学习中神经网络的泛化能力提供了新的思路。