The infrequent occurrence of overfit in deep neural networks is perplexing.
On the one hand, theory predicts that as models get larger they should
eventually become too specialized for a specific training set, with ensuing
decrease in generalization. In contrast, empirical results in image
classification indicate that increasing the training time of deep models or
using bigger models almost never hurts generalization. Is it because the way we
measure overfit is too limited? Here, we introduce a novel score for
quantifying overfit, which monitors the forgetting rate of deep models on
validation data. Presumably, this score indicates that even while
generalization improves overall, there are certain regions of the data space
where it deteriorates. When thus measured, we show that overfit can occur with
and without a decrease in validation accuracy, and may be more common than
previously appreciated. This observation may help to clarify the aforementioned
confusing picture. We use our observations to construct a new ensemble method,
based solely on the training history of a single network, which provides
significant improvement in performance without any additional cost in training
time. An extensive empirical evaluation with modern deep models shows our
method's utility on multiple datasets, neural networks architectures and
training schemes, both when training from scratch and when using pre-trained
networks in transfer learning. Notably, our method outperforms comparable
methods while being easier to implement and use, and further improves the
performance of competitive networks on Imagenet by 1\%.

在深度神经网络中，我们引入一种新的评分方法来衡量过拟合，该方法通过监测验证数据上的模型遗忘率，从而揭示了即使总体上泛化能力得到改善，仍存在数据空间中某些区域泛化能力退化的情况。基于这些观察，我们构建了一种新的集成方法，该方法仅基于单个网络的训练历史，在不增加额外训练时间的情况下显著提高了性能。在现代深度模型上进行的广泛实证评估显示了该方法在多个数据集、神经网络架构和训练方案中的实用性，无论是从头开始训练还是在迁移学习中使用预训练网络。值得注意的是，我们的方法在实现和使用上更加简单，并在 Imagenet 上改进了具有竞争力的网络 1% 的性能。

重拾被遗忘的知识：关于遗忘、过拟合和无需训练的深度神经网络集成

Relearning Forgotten Knowledge: on Forgetting, Overfit and Training-Free  Ensembles of DNNs

The method introduced in this paper aims at helping deep learning
practitioners faced with an overfit problem. The idea is to replace, in a
multi-branch network, the standard summation of parallel branches with a
stochastic affine combination. Applied to 3-branch residual networks,
shake-shake regularization improves on the best single shot published results
on CIFAR-10 and CIFAR-100 by reaching test errors of 2.86% and 15.85%.
Experiments on architectures without skip connections or Batch Normalization
show encouraging results and open the door to a large set of applications. Code
is available at this https URL

本文介绍了一种替换标准分支求和的随机仿射组合方法，用于改善深度学习中过拟合问题，结合掷骰子正则化可提高 CIFAR-10 和 CIFAR-100 测试准确度，对于跳跃连接或批量归一化的架构也有鼓舞人心的结果，具有广泛的应用前景。