The Neural Tangent Kernel (NTK) characterizes the behavior of infinitely-wide
neural networks trained under least squares loss by gradient descent. Recent
works also report that NTK regression can outperform finitely-wide neural
networks trained on small-scale datasets. However, the computational complexity
of kernel methods has limited its use in large-scale learning tasks. To
accelerate learning with NTK, we design a near input-sparsity time
approximation algorithm for NTK, by sketching the polynomial expansions of
arc-cosine kernels: our sketch for the convolutional counterpart of NTK (CNTK)
can transform any image using a linear runtime in the number of pixels.
Furthermore, we prove a spectral approximation guarantee for the NTK matrix, by
combining random features (based on leverage score sampling) of the arc-cosine
kernels with a sketching algorithm. We benchmark our methods on various
large-scale regression and classification tasks and show that a linear
regressor trained on our CNTK features matches the accuracy of exact CNTK on
CIFAR-10 dataset while achieving 150x speedup.

该研究提出了一种近似算法，旨在加速使用神经切向核的大规模学习任务，并结合随机特征，通过谱逼近保证精度。实验结果表明，其线性回归器可在 CIFAR-10 数据集上达到与全精度模型相当的准确度，同时提高了 150 倍的速度。