This work attempts to interpret modern deep (convolutional) networks from the
principles of rate reduction and (shift) invariant classification. We show that
the basic iterative gradient ascent scheme for optimizing the rate reduction of
learned features naturally leads to a multi-layer deep network, one iteration
per layer. The layered architectures, linear and nonlinear operators, and even
parameters of the network are all explicitly constructed layer-by-layer in a
forward propagation fashion by emulating the gradient scheme. All components of
this "white box" network have precise optimization, statistical, and geometric
interpretation. This principled framework also reveals and justifies the role
of multi-channel lifting and sparse coding in early stage of deep networks.
Moreover, all linear operators of the so-derived network naturally become
multi-channel convolutions when we enforce classification to be rigorously
shift-invariant. The derivation also indicates that such a convolutional
network is significantly more efficient to construct and learn in the spectral
domain. Our preliminary simulations and experiments indicate that so
constructed deep network can already learn a good discriminative representation
even without any back propagation training.

该研究尝试从速率减少和（移位）不变分类的原理解释现代深度（卷积）网络，并提供了一种基于梯度优化方案的前向传播方法来构建网络，其中包括层级结构，线性和非线性操作以及参数。此方法还揭示和证明了深度网络早期阶段的多通道升级和稀疏编码的作用，正式强制分类为严格移位不变时，所得到的网络的所有线性操作自然成为多通道卷积。同时，该卷积网络训练所需的成本更低，学习效率更高。初步模拟和实验表明，即使没有任何反向传导训练，基于该理论构建的深度网络已经可以学习良好的判别表示。