How can neural networks such as ResNet $\textit{efficiently}$ learn CIFAR-10 with test accuracy more than 96%, while other methods, especially kernel methods, fall far behind? Can we more provide theoretical justifications for this gap? There is an influential line of work relating neural networks to kernels in the over-parameterized regime, proving that they can learn certain concept class that is also learnable by kernels, with similar test error. Yet, can we show neural networks provably learn some concept class $\textit{better}$ than kernels? We answer this positively in the PAC learning language. We prove neural networks can efficiently learn a notable class of functions, including those defined by three-layer residual networks with smooth activations, without any distributional assumption. At the same time, we prove there are simple functions in this class that the test error obtained by neural networks can be $\textit{much smaller}$ than $\textit{any}$ "generic" kernel method, including neural tangent kernels, conjugate kernels, etc. The main intuition is that $\textit{multi-layer}$ neural networks can implicitly perform hierarchal learning using different layers, which reduces the sample complexity comparing to "one-shot" learning algorithms such as kernel methods.

该研究表明，多层神经网络通过不同层进行分层学习，缩小了与“单次学习”算法（如核方法）相比的样本复杂度，而这种思想与“反向特征校正”过程一起，又加深了深度网络的理论基础，同时也证明了ResNet相对于其他学习方法具有计算复杂度上的优势。

ResNet能高效地学习什么，超越核心卷积？