We present a theoretical approach to overcome the curse of dimensionality
using a neural computation algorithm which can be distributed across several
machines. Our modular distributed deep learning paradigm, termed \textit{neural
pathways}, can achieve arbitrary accuracy while only loading a small number of
parameters into GPU VRAM. Formally, we prove that for every error level
$\varepsilon>0$ and every Lipschitz function $f:[0,1]^n\to \mathbb{R}$, one can
construct a neural pathways model which uniformly approximates $f$ to
$\varepsilon$ accuracy over $[0,1]^n$ while only requiring networks of
$\mathcal{O}(\varepsilon^{-1})$ parameters to be loaded in memory and
$\mathcal{O}(\varepsilon^{-1}\log(\varepsilon^{-1}))$ to be loaded during the
forward pass. This improves the optimal bounds for traditional non-distributed
deep learning models, namely ReLU MLPs, which need
$\mathcal{O}(\varepsilon^{-n/2})$ parameters to achieve the same accuracy. The
only other available deep learning model that breaks the curse of
dimensionality is MLPs with super-expressive activation functions. However, we
demonstrate that these models have an infinite VC dimension, even with bounded
depth and width restrictions, unlike the neural pathways model. This implies
that only the latter generalizes. Our analysis is validated experimentally in
both regression and classification tasks, demonstrating that our model exhibits
superior performance compared to larger centralized benchmarks.

我们提出了一种理论方法，使用一种可以在多台机器上分布的神经计算算法来克服维度灾难。我们的模块化分布式深度学习范 paradigm 让我们能够在只加载少量参数到 GPU VRAM 的情况下实现任意的精度。我们从理论上证明，对于任意的错误水平 ε 和任何 Lipschitz 函数 f:[0,1]^n -> R，我们可以构建一个神经途径模型，在整个区间 [0,1]^n 上均匀地以 ε 精度逼近 f，只需加载大约 ε^-1 的参数到内存，并在前向传播过程中加载大约 ε^-1log (ε^-1) 的参数。这改进了传统非分布式深度学习模型（如 ReLU MLPs）达到相同精度所需要的参数的最优界限，后者需要大约 ε^-n/2 的参数。唯一能克服维度灾难的其他深度学习模型是具有超强表达能力的激活函数的 MLPs。然而，我们证明这些模型 VC 维是无穷大的，即使有有限的深度和宽度限制，而神经途径模型则不然。这意味着后者是唯一可以泛化的模型。我们的分析在回归和分类任务中通过实验证实，证明相比较较大的集中式基准模型，我们的模型表现出更好的性能。

分布式神经计算打破维度灾难

Breaking the Curse of Dimensionality with Distributed Neural Computation

Neural pathways as model explanations consist of a sparse set of neurons that
provide the same level of prediction performance as the whole model. Existing
methods primarily focus on accuracy and sparsity but the generated pathways may
offer limited interpretability thus fall short in explaining the model
behavior. In this paper, we suggest two interpretability criteria of neural
pathways: (i) same-class neural pathways should primarily consist of
class-relevant neurons; (ii) each instance's neural pathway sparsity should be
optimally determined. To this end, we propose a Generative Class-relevant
Neural Pathway (GEN-CNP) model that learns to predict the neural pathways from
the target model's feature maps. We propose to learn class-relevant information
from features of deep and shallow layers such that same-class neural pathways
exhibit high similarity. We further impose a faithfulness criterion for GEN-CNP
to generate pathways with instance-specific sparsity. We propose to transfer
the class-relevant neural pathways to explain samples of the same class and
show experimentally and qualitatively their faithfulness and interpretability.

神经通路作为模型解释的模型主要由一组稀疏的神经元组成，它们能够提供与整个模型相同水平的预测性能。本文提出了神经通路的两个可解释性准则：（一）同类神经通路主要由与类别相关的神经元组成；（二）每个实例的神经通路稀疏度应得到最佳确定。为此，我们提出了一种名为生成类相关神经通路（GEN-CNP）的模型，该模型能够从目标模型的特征图中学习预测神经通路。我们提出从深层和浅层特征中学习类相关信息，以使同类神经通路具有高度相似性。我们进一步对 GEN-CNP 施加忠实性准则，以生成具有实例特定稀疏性的路径。我们提出将类相关神经通路转化为解释同一类别样本，并通过实验证明了它们的忠实性和可解释性。

关于可迁移的神经通路的模型解释

On Model Explanations with Transferable Neural Pathways

The human brain can self-organize rich and diverse sparse neural pathways to
incrementally master hundreds of cognitive tasks. However, most existing
continual learning algorithms for deep artificial and spiking neural networks
are unable to adequately auto-regulate the limited resources in the network,
which leads to performance drop along with energy consumption rise as the
increase of tasks. In this paper, we propose a brain-inspired continual
learning algorithm with adaptive reorganization of neural pathways, which
employs Self-Organizing Regulation networks to reorganize the single and
limited Spiking Neural Network (SOR-SNN) into rich sparse neural pathways to
efficiently cope with incremental tasks. The proposed model demonstrates
consistent superiority in performance, energy consumption, and memory capacity
on diverse continual learning tasks ranging from child-like simple to complex
tasks, as well as on generalized CIFAR100 and ImageNet datasets. In particular,
the SOR-SNN model excels at learning more complex tasks as well as more tasks,
and is able to integrate the past learned knowledge with the information from
the current task, showing the backward transfer ability to facilitate the old
tasks. Meanwhile, the proposed model exhibits self-repairing ability to
irreversible damage and for pruned networks, could automatically allocate new
pathway from the retained network to recover memory for forgotten knowledge.

基于大脑启发的自适应神经路径重组的持续学习算法，在增量任务中通过自组织调控网络资源以有效应对多样化的认知任务，表现出卓越的性能、能量消耗和内存容量。具备学习更复杂任务、整合过去学习知识以及自我修复能力的特点。