End-to-end (E2E) training, optimizing the entire model through error
backpropagation, fundamentally supports the advancements of deep learning.
Despite its high performance, E2E training faces the problems of memory
consumption, parallel computing, and discrepancy with the functionalities of
the actual brain. Various alternative methods have been proposed to overcome
these difficulties; however, no one can yet match the performance of E2E
training, thereby falling short in practicality. Furthermore, there is no deep
understanding regarding differences in the trained model properties beyond the
performance gap. In this paper, we reconsider why E2E training demonstrates a
superior performance through a comparison with layer-wise training, a non-E2E
method that locally sets errors. On the basis of the observation that E2E
training has an advantage in propagating input information, we analyze the
information plane dynamics of intermediate representations based on the
Hilbert-Schmidt independence criterion (HSIC). The results of our normalized
HSIC value analysis reveal the E2E training ability to exhibit different
information dynamics across layers, in addition to efficient information
propagation. Furthermore, we show that this layer-role differentiation leads to
the final representation following the information bottleneck principle. It
suggests the need to consider the cooperative interactions between layers, not
just the final layer when analyzing the information bottleneck of deep
learning.

通过与逐层训练进行比较，本文重新考虑为什么端到端训练能够表现出卓越性能，并分析了中间表示的信息平面动态，揭示了端到端训练在不同层中展示不同信息动态的能力，以及表明层间的合作相互作用需要被考虑在深度学习的信息瓶颈分析中。

通过层级角色差异来诱导端到端训练中的信息瓶颈：与逐层训练进行比较的分析

End-to-End Training Induces Information Bottleneck through Layer-Role  Differentiation: A Comparative Analysis with Layer-wise Training

When using deep, multi-layered architectures to build generative models of
data, it is difficult to train all layers at once. We propose a layer-wise
training procedure admitting a performance guarantee compared to the global
optimum. It is based on an optimistic proxy of future performance, the best
latent marginal. We interpret auto-encoders in this setting as generative
models, by showing that they train a lower bound of this criterion. We test the
new learning procedure against a state of the art method (stacked RBMs), and
find it to improve performance. Both theory and experiments highlight the
importance, when training deep architectures, of using an inference model (from
data to hidden variables) richer than the generative model (from hidden
variables to data).

本文提出了一种基于乐观的概率方法进行逐层训练 DeepAutoEncoder 的过程，可以提高深度架构训练的性能，实验结果表明使用比生成模型更丰富的推理模型可以有效地提高性能。