Highly overparametrized neural networks can display curiously strong generalization performance - a phenomenon that has recently garnered a wealth of theoretical and empirical research in order to better understand it. In contrast to most previous work, which typically considers the performance as a function of the model size, in this paper we empirically study the generalization performance as the size of the training set varies over multiple orders of magnitude. These systematic experiments lead to some interesting and potentially very useful observations; perhaps most notably that training on smaller subsets of the data can lead to more reliable model selection decisions whilst simultaneously enjoying smaller computational costs. Our experiments furthermore allow us to estimate Minimum Description Lengths for common datasets given modern neural network architectures, thereby paving the way for principled model selection taking into account Occams-razor.

本文通过实验证明，相比于之前研究中通常考虑模型大小因素对性能的影响，训练数据集的大小对模型性能的影响同样显著，实验结果发现，训练更小的数据子集可以带来更可靠的模型选择决策同时降低计算成本，并能估算出常见数据集在现代神经网络结构下的最小描述长度，为采用奥卡姆剃刀原则进行负责的模型选择提供了可能性。

小数据，大决策：小数据情境下的模型选择