This paper presents a new family of backpropagation-free neural
architectures, Gated Linear Networks (GLNs). What distinguishes GLNs from
contemporary neural networks is the distributed and local nature of their
credit assignment mechanism; each neuron directly predicts the target, forgoing
the ability to learn feature representations in favor of rapid online learning.
Individual neurons can model nonlinear functions via the use of data-dependent
gating in conjunction with online convex optimization. We show that this
architecture gives rise to universal learning capabilities in the limit, with
effective model capacity increasing as a function of network size in a manner
comparable with deep ReLU networks. Furthermore, we demonstrate that the GLN
learning mechanism possesses extraordinary resilience to catastrophic
forgetting, performing comparably to a MLP with dropout and Elastic Weight
Consolidation on standard benchmarks. These desirable theoretical and empirical
properties position GLNs as a complementary technique to contemporary offline
deep learning methods.

该论文介绍了一种新的神经网络结构，Gated Linear Networks (GLNs)，其区别于当代神经网络的地方是，其信用分配机制具有分布式和本地化特性；每个神经元都直接预测目标，放弃了学习特征表示的能力，而选择快速在线学习。使用数据相关门控和在线凸优化，个别神经元可以建模非线性函数。该论文证明了该架构在极限情况下具有普遍的学习能力，并且有效的模型容量随着网络大小的增加而增加，与深度 ReLU 网络相似。此外，该论文证明 GLN 学习机制具有非常强的抗遗忘能力，在标准基准测试中表现与带有 dropout 和 Elastic Weight Consolidation 的 MLP 相当。这些理论和实证特性将 GLNs 定位为当代离线深度学习方法的补充技术。