Backpropagation (BP) is the cornerstone of deep learning, but its reliance on global gradient synchronization limits scalability and imposes significant memory overhead. We propose Stochastic Variational Propagation (SVP), a scalable alternative that reframes training as hierarchical variational inference. SVP treats layer activations as latent variables and optimizes local Evidence Lower Bounds (ELBOs), enabling independent, local updates while preserving global coherence. However, directly applying KL divergence in layer-wise ELBOs risks inter-layer's representation collapse due to excessive compression. To prevent this, SVP projects activations into low-dimensional spaces via fixed random matrices, ensuring information preservation and representational diversity. Combined with a feature alignment loss for inter-layer consistency, SVP achieves competitive accuracy with BP across diverse architectures (MLPs, CNNs, Transformers) and datasets (MNIST to ImageNet), reduces memory usage by up to 4x, and significantly improves scalability. More broadly, SVP introduces a probabilistic perspective to deep representation learning, opening pathways toward more modular and interpretable neural network design.

本研究解决了反向传播在深度学习中的全球梯度同步限制与显著内存开销问题。我们提出的随机变分传播（SVP）通过将训练重构为分层变分推断，允许独立的本地更新，同时保持全局一致性，最终在各种架构和数据集上实现了与反向传播相媲美的精度，内存使用减少了多达四倍，显著提高了可扩展性。

随机变分传播：局部、可扩展和高效的替代反向传播方法