We study how abstract representations emerge in a Deep Belief Network (DBN) trained on benchmark datasets. Our analysis targets the principles of learning in the early stages of information processing, starting from the "primordial soup" of the under-sampling regime. As the data is processed by deeper and deeper layers, features are detected and removed, transferring more and more "context-invariant" information to deeper layers. We show that the representation approaches an universal model -- the Hierarchical Feature Model (HFM) -- determined by the principle of maximal relevance. Relevance quantifies the uncertainty on the model of the data, thus suggesting that "meaning" -- i.e. syntactic information -- is that part of the data which is not yet captured by a model. Our analysis shows that shallow layers are well described by pairwise Ising models, which provide a representation of the data in terms of generic, low order features. We also show that plasticity increases with depth, in a similar way as it does in the brain. These findings suggest that DBNs are capable of extracting a hierarchy of features from the data which is consistent with the principle of maximal relevance.

研究了深度置信网络在基准数据集上的训练中抽象表示如何出现。分析表明，使用越来越深的层次处理数据，能够检测和消除特征，并将越来越多的'上下文无关'信息传递给更深的层次。研究表明浅层可由成对的Ising模型描述，这些模型提供了一种用通用的低阶特征来表示数据的方式。结果还显示，可塑性随深度增加而增加，类似于大脑。这些发现表明深度置信网络能够从数据中提取与最大相关性原则一致的特征层次结构。

深度神经网络中的抽象的统计特征