We introduce rainbow networks as a probabilistic model of trained deep neural
networks. The model cascades random feature maps whose weight distributions are
learned. It assumes that dependencies between weights at different layers are
reduced to rotations which align the input activations. Neuron weights within a
layer are independent after this alignment. Their activations define kernels
which become deterministic in the infinite-width limit. This is verified
numerically for ResNets trained on the ImageNet dataset. We also show that the
learned weight distributions have low-rank covariances. Rainbow networks thus
alternate between linear dimension reductions and non-linear high-dimensional
embeddings with white random features. Gaussian rainbow networks are defined
with Gaussian weight distributions. These models are validated numerically on
image classification on the CIFAR-10 dataset, with wavelet scattering networks.
We further show that during training, SGD updates the weight covariances while
mostly preserving the Gaussian initialization.

引入彩虹网络作为训练的深度神经网络的概率模型，通过学习随机特征图的权重分布，并假设不同层之间的权重依赖被缩减为使输入激活对齐的旋转，介绍了其在 ImageNet 数据集上的验证和 Gaussian 权重分布的 Rainbow 网络的验证。