We introduce the unbounded depth neural network (UDN), an infinitely deep
probabilistic model that adapts its complexity to the training data. The UDN
contains an infinite sequence of hidden layers and places an unbounded prior on
a truncation L, the layer from which it produces its data. Given a dataset of
observations, the posterior UDN provides a conditional distribution of both the
parameters of the infinite neural network and its truncation. We develop a
novel variational inference algorithm to approximate this posterior, optimizing
a distribution of the neural network weights and of the truncation depth L, and
without any upper limit on L. To this end, the variational family has a special
structure: it models neural network weights of arbitrary depth, and it
dynamically creates or removes free variational parameters as its distribution
of the truncation is optimized. (Unlike heuristic approaches to model search,
it is solely through gradient-based optimization that this algorithm explores
the space of truncations.) We study the UDN on real and synthetic data. We find
that the UDN adapts its posterior depth to the dataset complexity; it
outperforms standard neural networks of similar computational complexity; and
it outperforms other approaches to infinite-depth neural networks.

介绍了无限深度神经网络 (UDN)，这是一种无穷深的概率模型，可以根据训练数据自适应其复杂性。它包含一个无限序列的隐藏层，并对截止层数形成一个无界先验。通过开发一种新的变分推断算法来逼近其后验分布，优化神经网络权重和截止层数的分布，该算法探索了截止层数的空间。在真实和合成数据上验证了 UDN 的适应性，以及其与标准神经网络和其他无限深度神经网络方法的性能比较。