Learning, taking into account full distribution of the data, referred to as
generative, is not feasible with deep neural networks (DNNs) because they model
only the conditional distribution of the outputs given the inputs. Current
solutions are either based on joint probability models facing difficult
estimation problems or learn two separate networks, mapping inputs to outputs
(recognition) and vice-versa (generation). We propose an intermediate approach.
First, we show that forward computation in DNNs with logistic sigmoid
activations corresponds to a simplified approximate Bayesian inference in a
directed probabilistic multi-layer model. This connection allows to interpret
DNN as a probabilistic model of the output and all hidden units given the
input. Second, we propose that in order for the recognition and generation
networks to be more consistent with the joint model of the data, weights of the
recognition and generator network should be related by transposition. We
demonstrate in a tentative experiment that such a coupled pair can be learned
generatively, modelling the full distribution of the data, and has enough
capacity to perform well in both recognition and generation.

本文的研究旨在深度神经网络的生成式学习问题上提出一种中间方法，通过前向计算得出一个简化版的贝叶斯概率多层模型，并且运用转置的方法使得识别和生成网络更加符合数据的联合概率模型。通过实验发现，这种联合对组的生成对模型足够强大，可以很好地进行识别和生成任务。