The Manual labeling of data is and will remain a costly endeavor. For this
reason, semi-supervised learning remains a topic of practical importance. The
recently proposed Ladder Network is one such approach that has proven to be
very successful. In addition to the supervised objective, the Ladder Network
also adds an unsupervised objective corresponding to the reconstruction costs
of a stack of denoising autoencoders. Although the empirical results are
impressive, the Ladder Network has many components intertwined, whose
contributions are not obvious in such a complex architecture. In order to help
elucidate and disentangle the different ingredients in the Ladder Network
recipe, this paper presents an extensive experimental investigation of variants
of the Ladder Network in which we replace or remove individual components to
gain more insight into their relative importance. We find that all of the
components are necessary for achieving optimal performance, but they do not
contribute equally. For semi-supervised tasks, we conclude that the most
important contribution is made by the lateral connection, followed by the
application of noise, and finally the choice of what we refer to as the
`combinator function' in the decoder path. We also find that as the number of
labeled training examples increases, the lateral connections and reconstruction
criterion become less important, with most of the improvement in generalization
being due to the injection of noise in each layer. Furthermore, we present a
new type of combinator function that outperforms the original design in both
fully- and semi-supervised tasks, reducing record test error rates on
Permutation-Invariant MNIST to 0.57% for the supervised setting, and to 0.97%
and 1.0% for semi-supervised settings with 1000 and 100 labeled examples
respectively.

本文研究了一种神经网络架构 Ladder Network，通过剔除或替换部分组件，探究了其各个组成部分的贡献及重要性，并提出了新型组合函数，在全监督和半监督任务中均取得了较好的性能表现。

分解梯度网络架构

Deconstructing the Ladder Network Architecture

Suitable lateral connections between encoder and decoder are shown to allow
higher layers of a denoising autoencoder (dAE) to focus on invariant
representations. In regular autoencoders, detailed information needs to be
carried through the highest layers but lateral connections from encoder to
decoder relieve this pressure. It is shown that abstract invariant features can
be translated to detailed reconstructions when invariant features are allowed
to modulate the strength of the lateral connection. Three dAE structures with
modulated and additive lateral connections, and without lateral connections
were compared in experiments using real-world images. The experiments verify
that adding modulated lateral connections to the model 1) improves the accuracy
of the probability model for inputs, as measured by denoising performance; 2)
results in representations whose degree of invariance grows faster towards the
higher layers; and 3) supports the formation of diverse invariant poolings.

研究表明，编码器和解码器之间的适当侧向连接可以使去噪自编码器（dAE）的更高层专注于不变表示。添加被允许调制侧向连接的不变特征可以将抽象不变特征翻译为详细的重构，并支持形成不同的不变池。通过在真实世界图像上进行实验，发现添加调制的侧向连接到模型中可以提高模型对输入的概率模型准确性，产生的表示不变度在更高层次上增加得更快，并支持形成多样的不变池塘。