Common infinite-width architectures such as Neural Tangent Kernels (NTKs)
have historically shown weak performance compared to finite models. This has
been attributed to the absence of feature learning. We show that this is not
the case. In fact, we show that infinite width NTK models are able to access
richer features than finite models by selecting relevant subfeatures from their
(infinite) feature vector. In fact, we show experimentally that NTKs
under-perform traditional finite models even when feature learning is
artificially disabled. Instead, weak performance is due to the fact that
existing constructions depend on weak optimizers like SGD. We provide an
infinite width limit based on ADAM-like learning dynamics and demonstrate
empirically that the resulting models erase this performance gap.

无限宽度架构，如神经切向核 (NTK) 在过去显示出与有限模型相比较弱的性能。我们证明这并非如此，实际上，我们展示无限宽度的 NTK 模型能够通过从无限特征向量中选择相关子特征来访问更丰富的特征。事实上，我们实验性地表明，即使在关闭特征学习的情况下，NTK 仍不如传统的有限模型性能好。相反，性能不佳是因为现有的构造依赖于像 SGD 这样的弱优化器。我们提供了基于类似于 ADAM 的学习动力学的无限宽度极限，并通过实验证明生成的模型消除了性能差距。