The utilization of residual learning has become widespread in deep and
scalable neural nets. However, the fundamental principles that contribute to
the success of residual learning remain elusive, thus hindering effective
training of plain nets with depth scalability. In this paper, we peek behind
the curtains of residual learning by uncovering the "dissipating inputs"
phenomenon that leads to convergence failure in plain neural nets: the input is
gradually compromised through plain layers due to non-linearities, resulting in
challenges of learning feature representations. We theoretically demonstrate
how plain neural nets degenerate the input to random noise and emphasize the
significance of a residual connection that maintains a better lower bound of
surviving neurons as a solution. With our theoretical discoveries, we propose
"The Plain Neural Net Hypothesis" (PNNH) that identifies the internal path
across non-linear layers as the most critical part in residual learning, and
establishes a paradigm to support the training of deep plain neural nets devoid
of residual connections. We thoroughly evaluate PNNH-enabled CNN architectures
and Transformers on popular vision benchmarks, showing on-par accuracy, up to
0.3% higher training throughput, and 2x better parameter efficiency compared to
ResNets and vision Transformers.

通过发现神经网络中的 “消散输入” 现象，我们揭示了残差学习背后的基本原理，从而解决了深度可扩展的普通神经网络训练中收敛失败的挑战，并提出了 “普通神经网络假设”（PNNH）来支持无残差连接的深度普通神经网络训练，通过在流行的视觉基准测试中对 PNNH 启用的 CNN 架构和 Transformers 进行全面评估，并展示了与 ResNets 和视觉 Transformers 相比的相当准确性、高达 0.3% 更高的训练吞吐量和 2 倍更好的参数效率。