Traditional end-to-end (E2E) training of deep networks necessitates storing
intermediate activations for back-propagation, resulting in a large memory
footprint on GPUs and restricted model parallelization. As an alternative,
greedy local learning partitions the network into gradient-isolated modules and
trains supervisely based on local preliminary losses, thereby providing
asynchronous and parallel training methods that substantially reduce memory
cost. However, empirical experiments reveal that as the number of segmentations
of the gradient-isolated module increases, the performance of the local
learning scheme degrades substantially, severely limiting its expansibility. To
avoid this issue, we theoretically analyze the greedy local learning from the
standpoint of information theory and propose a ContSup scheme, which
incorporates context supply between isolated modules to compensate for
information loss. Experiments on benchmark datasets (i.e. CIFAR, SVHN, STL-10)
achieve SOTA results and indicate that our proposed method can significantly
improve the performance of greedy local learning with minimal memory and
computational overhead, allowing for the boost of the number of isolated
modules. Our codes are available at this https URL

从信息论的角度出发，我们在贪婪的局部学习中提出了一种 ContSup 方案，它将隔离模块之间的上下文补充作为补偿信息损失的手段。在基准数据集（即 CIFAR、SVHN、STL-10）上的实验证明，我们提出的方法能够显著提高贪婪的局部学习性能，并且只带来最小的内存和计算开销，可以增加隔离模块的数量。