A neural architecture with randomly initialized weights, in the infinite
width limit, is equivalent to a Gaussian Random Field whose covariance function
is the so-called Neural Network Gaussian Process kernel (NNGP). We prove that a
reproducing kernel Hilbert space (RKHS) defined by the NNGP contains only
functions that can be approximated by the architecture. To achieve a certain
approximation error the required number of neurons in each layer is defined by
the RKHS norm of the target function. Moreover, the approximation can be
constructed from a supervised dataset by a random multi-layer representation of
an input vector, together with training of the last layer's weights.
For a 2-layer NN and a domain equal to an $n-1$-dimensional sphere in
${\mathbb R}^n$, we compare the number of neurons required by Barron's theorem
and by the multi-layer features construction. We show that if eigenvalues of
the integral operator of the NNGP decay slower than $k^{-n-\frac{2}{3}}$ where
$k$ is an order of an eigenvalue, then our theorem guarantees a more succinct
neural network approximation than Barron's theorem. We also make some
computational experiments to verify our theoretical findings. Our experiments
show that realistic neural networks easily learn target functions even when
both theorems do not give any guarantees.

神经网络架构、随机初始化权重、神经网络高斯过程核、再生核希尔伯特空间、逼近误差是该研究论文的关键词，论文提出了一种在无限宽度限制下具有随机初始化权重的神经网络架构，它等价于一个具有高斯随机场协方差函数的神经网络高斯过程核，同时证明了该神经网络架构可以逼近由该核定义的再生核希尔伯特空间中的函数。实验结果验证了该理论发现的可行性。

多层随机特征和神经网络的逼近能力

Multi-layer random features and the approximation power of neural  networks

Dataset distillation compresses large datasets into smaller synthetic
coresets which retain performance with the aim of reducing the storage and
computational burden of processing the entire dataset. Today's best-performing
algorithm, \textit{Kernel Inducing Points} (KIP), which makes use of the
correspondence between infinite-width neural networks and kernel-ridge
regression, is prohibitively slow due to the exact computation of the neural
tangent kernel matrix, scaling $O(|S|^2)$, with $|S|$ being the coreset size.
To improve this, we propose a novel algorithm that uses a random feature
approximation (RFA) of the Neural Network Gaussian Process (NNGP) kernel, which
reduces the kernel matrix computation to $O(|S|)$. Our algorithm provides at
least a 100-fold speedup over KIP and can run on a single GPU. Our new method,
termed an RFA Distillation (RFAD), performs competitively with KIP and other
dataset condensation algorithms in accuracy over a range of large-scale
datasets, both in kernel regression and finite-width network training. We
demonstrate the effectiveness of our approach on tasks involving model
interpretability and privacy preservation.

提出了一种名为 RFAD 的基于随机特征近似的数据集蒸馏算法，该算法能够在维持较高准确性的同时，大幅加速了现有的数据集压缩算法，能够应用于大规模数据集上，并适用于如模型解释和隐私保护等任务。