The backpropagation algorithm remains the dominant and most successful method
for training deep neural networks (DNNs). At the same time, training DNNs at
scale comes at a significant computational cost and therefore a high carbon
footprint. Converging evidence suggests that input decorrelation may speed up
deep learning. However, to date, this has not yet translated into substantial
improvements in training efficiency in large-scale DNNs. This is mainly caused
by the challenge of enforcing fast and stable network-wide decorrelation. Here,
we show for the first time that much more efficient training of very deep
neural networks using decorrelated backpropagation is feasible. To achieve this
goal we made use of a novel algorithm which induces network-wide input
decorrelation using minimal computational overhead. By combining this algorithm
with careful optimizations, we obtain a more than two-fold speed-up and higher
test accuracy compared to backpropagation when training a 18-layer deep
residual network. This demonstrates that decorrelation provides exciting
prospects for efficient deep learning at scale.

使用去相关反向传播算法可以显著提高大规模深度神经网络的训练效率，并且可以获得更高的测试准确率。