Generalization performance of classifiers in deep learning has recently
become a subject of intense study. Deep models, typically over-parametrized,
tend to fit the training data exactly. Despite this "overfitting", they perform
well on test data, a phenomenon not yet fully understood.
The first point of our paper is that strong performance of overfitted
classifiers is not a unique feature of deep learning. Using six real-world and
two synthetic datasets, we establish experimentally that kernel machines
trained to have zero classification or near zero regression error perform very
well on test data, even when the labels are corrupted with a high level of
noise. We proceed to give a lower bound on the norm of zero loss solutions for
smooth kernels, showing that they increase nearly exponentially with data size.
We point out that this is difficult to reconcile with the existing
generalization bounds. Moreover, none of the bounds produce non-trivial results
for interpolating solutions.
Second, we show experimentally that (non-smooth) Laplacian kernels easily fit
random labels, a finding that parallels results for ReLU neural networks. In
contrast, fitting noisy data requires many more epochs for smooth Gaussian
kernels. Similar performance of overfitted Laplacian and Gaussian classifiers
on test, suggests that generalization is tied to the properties of the kernel
function rather than the optimization process.
Certain key phenomena of deep learning are manifested similarly in kernel
methods in the modern "overfitted" regime. The combination of the experimental
and theoretical results presented in this paper indicates a need for new
theoretical ideas for understanding properties of classical kernel methods. We
argue that progress on understanding deep learning will be difficult until more
tractable "shallow" kernel methods are better understood.

研究表明深度学习和浅层核方法在过拟合问题上表现相似，而不是泛化性能高的唯一因素，因此需要更深入地研究浅层核方法，以更好地理解深度学习。