When training neural networks for classification tasks with backpropagation,
parameters are updated on every trial, even if the sample is classified
correctly. In contrast, humans concentrate their learning effort on errors.
Inspired by human learning, we introduce lazy learning, which only learns on
incorrect samples. Lazy learning can be implemented in a few lines of code and
requires no hyperparameter tuning. Lazy learning achieves state-of-the-art
performance and is particularly suited when datasets are large. For instance,
it reaches 99.2% test accuracy on Extended MNIST using a single-layer MLP, and
does so 7.6x faster than a matched backprop network

本文介绍了一种受人类学习模式启发的方法：懒惰学习，通过只在分错的样本上进行学习，可实现一些分类任务的更高准确度，并且不需要超参数调整。

惰性学习：一种受生物启发的可塑性规则，用于快速、高效的突触可塑性

Lazy learning: a biologically-inspired plasticity rule for fast and energy efficient synaptic plasticity

Deep neural networks can achieve remarkable generalization performances while
interpolating the training data perfectly. Rather than the U-curve emblematic
of the bias-variance trade-off, their test error often follows a "double
descent" - a mark of the beneficial role of overparametrization. In this work,
we develop a quantitative theory for this phenomenon in the so-called lazy
learning regime of neural networks, by considering the problem of learning a
high-dimensional function with random features regression. We obtain a precise
asymptotic expression for the bias-variance decomposition of the test error,
and show that the bias displays a phase transition at the interpolation
threshold, beyond which it remains constant. We disentangle the variances
stemming from the sampling of the dataset, from the additive noise corrupting
the labels, and from the initialization of the weights. Following up on Geiger
et al. 2019, we first show that the latter two contributions are the crux of
the double descent: they lead to the overfitting peak at the interpolation
threshold and to the decay of the test error upon overparametrization. We then
quantify how they are suppressed by ensemble averaging the outputs of K
independently initialized estimators. When K is sent to infinity, the test
error remains constant beyond the interpolation threshold. We further compare
the effects of overparametrizing, ensembling and regularizing. Finally, we
present numerical experiments on classic deep learning setups to show that our
results hold qualitatively in realistic lazy learning scenarios.

研究发现，通过过度参数化，深度神经网络能够在插值训练数据的同时实现卓越的泛化性能，并且在测试误差上具有双下降现象，该现象可以通过集成平均估计器进行抑制。