Training a neural network using backpropagation algorithm requires passing
error gradients sequentially through the network. The backward locking prevents
us from updating network layers in parallel and fully leveraging the computing
resources. Recently, there are several works trying to decouple and parallelize
the backpropagation algorithm. However, all of them suffer from severe accuracy
loss or memory explosion when the neural network is deep. To address these
challenging issues, we propose a novel parallel-objective formulation for the
objective function of the neural network. After that, we introduce features
replay algorithm and prove that it is guaranteed to converge to critical points
for the non-convex problem under certain conditions. Finally, we apply our
method to training deep convolutional neural networks, and the experimental
results show that the proposed method achieves {faster} convergence, {lower}
memory consumption, and {better} generalization error than compared methods.

通过提出一种新的并行目标形式及引入特征重放算法，本文成功地解决了传统反向传播算法无法平行更新网络层的问题，并在实际中实现了更快的收敛速度、更低的内存消耗和更好的泛化性能。