We present the checkpoint ensembles method that can learn ensemble models on
a single training process. Although checkpoint ensembles can be applied to any
parametric iterative learning technique, here we focus on neural networks.
Neural networks' composable and simple neurons make it possible to capture many
individual and interaction effects among features. However, small sample sizes
and sampling noise may result in patterns in the training data that are not
representative of the true relationship between the features and the outcome.
As a solution, regularization during training is often used (e.g. dropout).
However, regularization is no panacea -- it does not perfectly address
overfitting. Even with methods like dropout, two methodologies are commonly
used in practice. First is to utilize a validation set independent to the
training set as a way to decide when to stop training. Second is to use
ensemble methods to further reduce overfitting and take advantage of local
optima (i.e. averaging over the predictions of several models). In this paper,
we explore checkpoint ensembles -- a simple technique that combines these two
ideas in one training process. Checkpoint ensembles improve performance by
averaging the predictions from "checkpoints" of the best models within single
training process. We use three real-world data sets -- text, image, and
electronic health record data -- using three prediction models: a vanilla
neural network, a convolutional neural network, and a long short term memory
network to show that checkpoint ensembles outperform existing methods: a method
that selects a model by minimum validation score, and two methods that average
models by weights. Our results also show that checkpoint ensembles capture a
portion of the performance gains that traditional ensembles provide.

本文介绍了一种名为 Checkpoint Ensembles 的技术，它能够在单个训练过程中生成集成模型，使用此技术可以解决神经网络样本数据过小、采样噪声问题导致的训练数据误差。该方法通过检查点 ensemble 的方式融合了 validation set 和 ensemble models，实验证明 Checkpoint Ensembles 的表现优于其他方法。