A continuing mystery in understanding the empirical success of deep neural
networks is their ability to achieve zero training error and generalize well,
even when the training data is noisy and there are more parameters than data
points. We investigate this overparameterized regime in linear regression,
where all solutions that minimize training error interpolate the data,
including noise. We characterize the fundamental generalization (mean-squared)
error of any interpolating solution in the presence of noise, and show that
this error decays to zero with the number of features. Thus,
overparameterization can be explicitly beneficial in ensuring harmless
interpolation of noise. We discuss two root causes for poor generalization that
are complementary in nature -- signal "bleeding" into a large number of alias
features, and overfitting of noise by parsimonious feature selectors. For the
sparse linear model with noise, we provide a hybrid interpolating scheme that
mitigates both these issues and achieves order-optimal MSE over all possible
interpolating solutions.

本研究探讨了深度神经网络在训练数据含有噪声且参数个数超过数据点个数时，仍能够实现零训练误差且具有泛化能力的机制，并阐述了过拟合和特征选择不佳对泛化能力的影响。

回归中的无害噪声数据插值

Harmless interpolation of noisy data in regression

Artificial neural networks are most commonly trained with the
back-propagation algorithm, where the gradient for learning is provided by
back-propagating the error, layer by layer, from the output layer to the hidden
layers. A recently discovered method called feedback-alignment shows that the
weights used for propagating the error backward don't have to be symmetric with
the weights used for propagation the activation forward. In fact, random
feedback weights work evenly well, because the network learns how to make the
feedback useful. In this work, the feedback alignment principle is used for
training hidden layers more independently from the rest of the network, and
from a zero initial condition. The error is propagated through fixed random
feedback connections directly from the output layer to each hidden layer. This
simple method is able to achieve zero training error even in convolutional
networks and very deep networks, completely without error back-propagation. The
method is a step towards biologically plausible machine learning because the
error signal is almost local, and no symmetric or reciprocal weights are
required. Experiments show that the test performance on MNIST and CIFAR is
almost as good as those obtained with back-propagation for fully connected
networks. If combined with dropout, the method achieves 1.45% error on the
permutation invariant MNIST task.

使用反馈定向方法进行神经网络训练的简单方法能够在卷积网络和深度网络中实现零训练误差，而不需要成对的权重，是迈向生物可行机器学习的一步。