A long-standing obstacle to progress in deep learning is the problem of
vanishing and exploding gradients. Although, the problem has largely been
overcome via carefully constructed initializations and batch normalization,
architectures incorporating skip-connections such as highway and resnets
perform much better than standard feedforward architectures despite well-chosen
initialization and batch normalization. In this paper, we identify the
shattered gradients problem. Specifically, we show that the correlation between
gradients in standard feedforward networks decays exponentially with depth
resulting in gradients that resemble white noise whereas, in contrast, the
gradients in architectures with skip-connections are far more resistant to
shattering, decaying sublinearly. Detailed empirical evidence is presented in
support of the analysis, on both fully-connected networks and convnets.
Finally, we present a new "looks linear" (LL) initialization that prevents
shattering, with preliminary experiments showing the new initialization allows
to train very deep networks without the addition of skip-connections.

本文提出了 “破碎梯度” 问题，指出标准前馈网络中梯度之间的相关性随着深度呈指数衰减，导致梯度类似于白噪声；相比之下，具有跳跃连接的体系结构更加抗干扰，梯度呈次线性降低。此外，本文还提出了一种新的 “线性外观”(LL) 初始化方法，可以避免破碎现象，并证明其可以使深度网络更好地进行训练。