It has been believed that stochastic feedforward neural networks (SFNNs) have
several advantages beyond deterministic deep neural networks (DNNs): they have
more expressive power allowing multi-modal mappings and regularize better due
to their stochastic nature. However, training large-scale SFNN is notoriously
harder. In this paper, we aim at developing efficient training methods for
SFNN, in particular using known architectures and pre-trained parameters of
DNN. To this end, we propose a new intermediate stochastic model, called
Simplified-SFNN, which can be built upon any baseline DNNand approximates
certain SFNN by simplifying its upper latent units above stochastic ones. The
main novelty of our approach is in establishing the connection between three
models, i.e., DNN->Simplified-SFNN->SFNN, which naturally leads to an efficient
training procedure of the stochastic models utilizing pre-trained parameters of
DNN. Using several popular DNNs, we show how they can be effectively
transferred to the corresponding stochastic models for both multi-modal and
classification tasks on MNIST, TFD, CASIA, CIFAR-10, CIFAR-100 and SVHN
datasets. In particular, we train a stochastic model of 28 layers and 36
million parameters, where training such a large-scale stochastic network is
significantly challenging without using Simplified-SFNN

本文提出了一种新的中间随机模型，称为简化 SFNN，该模型可以建立在任何基线 DNN 之上，并通过简化其上部随机单元来近似某些 SFNN。我们将 DNN-> Simplified-SFNN-> SFNN 三个模型联系在一起，从而自然地引导随机模型的高效训练过程，利用 DNN 的预训练参数。使用多个流行的 DNN，我们展示了如何将它们有效地转移到相应的随机模型，以进行 MNIST，TFD，CASIA，CIFAR-10，CIFAR-100 和 SVHN 数据集的多模态和分类任务。