We show that deep narrow Boltzmann machines are universal approximators of
probability distributions on the activities of their visible units, provided
they have sufficiently many hidden layers, each containing the same number of
units as the visible layer. We show that, within certain parameter domains,
deep Boltzmann machines can be studied as feedforward networks. We provide
upper and lower bounds on the sufficient depth and width of universal
approximators. These results settle various intuitions regarding undirected
networks and, in particular, they show that deep narrow Boltzmann machines are
at least as compact universal approximators as narrow sigmoid belief networks
and restricted Boltzmann machines, with respect to the currently available
bounds for those models.

该研究表明，深窄玻尔兹曼机是可视单元活动上概率分布的普适逼近器，如果它们有足够多的隐藏层，每个层次含有与可见层相同数量的单元。该研究提供了普适逼近器所需的深度和宽度的上下界，并且解决了有关无向网络的各种直觉问题，并且特别表明，与狭窄的 S 型信念网络和受限玻尔兹曼机相比，深窄玻尔兹曼机至少与当前可用于这些模型的限制相一致。

深度窄玻尔兹曼机是普适逼近器

Deep Narrow Boltzmann Machines are Universal Approximators

Our proposed deeply-supervised nets (DSN) method simultaneously minimizes
classification error while making the learning process of hidden layers direct
and transparent. We make an attempt to boost the classification performance by
studying a new formulation in deep networks. Three aspects in convolutional
neural networks (CNN) style architectures are being looked at: (1) transparency
of the intermediate layers to the overall classification; (2)
discriminativeness and robustness of learned features, especially in the early
layers; (3) effectiveness in training due to the presence of the exploding and
vanishing gradients. We introduce "companion objective" to the individual
hidden layers, in addition to the overall objective at the output layer (a
different strategy to layer-wise pre-training). We extend techniques from
stochastic gradient methods to analyze our algorithm. The advantage of our
method is evident and our experimental result on benchmark datasets shows
significant performance gain over existing methods (e.g. all state-of-the-art
results on MNIST, CIFAR-10, CIFAR-100, and SVHN).

本文提出了一种名为深度监督网络（DSN）的方法，该方法同时最小化分类错误并使隐藏层的学习过程变得直接和透明；通过研究深层网络中的新表述，从卷积神经网络结构的三个方面来提高分类性能：中间层到整体分类的透明度、学习特征的区分能力及健壮性（特别是在早期层），以及存在爆炸和消失梯度的有效训练。我们在个别隐藏层中引入 “伴随目标”，并结合随机梯度方法来分析算法。实验结果表明，相对于现有方法，我们的方法具有显著的性能提升（例如在 MNIST，CIFAR-10，CIFAR-100 和 SVHN 等基准数据集上的表现均优于现有技术的最新水平）。

深度监督网络

Deeply-Supervised Nets

Good old on-line back-propagation for plain multi-layer perceptrons yields a
very low 0.35% error rate on the famous MNIST handwritten digits benchmark. All
we need to achieve this best result so far are many hidden layers, many neurons
per layer, numerous deformed training images, and graphics cards to greatly
speed up learning.

使用在线反向传播算法的多层感知机，通过增加隐藏层数、每层神经元的数量，并且利用数百个变形训练图片和图形加速卡等技术，仅有 0.35% 的误差率就可以在著名的 MNIST 手写数字基准测试上取得最佳结果。