Current state-of-the-art deep networks are all powered by backpropagation. In
this paper, we explore alternatives to full backpropagation in the form of
blockwise learning rules, leveraging the latest developments in self-supervised
learning. We show that a blockwise pretraining procedure consisting of training
independently the 4 main blocks of layers of a ResNet-50 with Barlow Twins'
loss function at each block performs almost as well as end-to-end
backpropagation on ImageNet: a linear probe trained on top of our blockwise
pretrained model obtains a top-1 classification accuracy of 70.48%, only 1.1%
below the accuracy of an end-to-end pretrained network (71.57% accuracy). We
perform extensive experiments to understand the impact of different components
within our method and explore a variety of adaptations of self-supervised
learning to the blockwise paradigm, building an exhaustive understanding of the
critical avenues for scaling local learning rules to large networks, with
implications ranging from hardware design to neuroscience.

该论文中，我们使用最新的自监督学习技术，探索了全面反向传播的替代方法，即以块为单位的学习规则。我们证明了一个块状预训练程序，由 ResNet-50 的 4 个主要层块独立训练并使用 Barlow Twins' loss 函数，在 ImageNet 数据集上表现几乎与端到端反向传播相同，达到了 70.48% 的 top-1 分类准确率。