Ensemble Learning is an effective method for improving generalization in
machine learning. However, as state-of-the-art neural networks grow larger, the
computational cost associated with training several independent networks
becomes expensive. We introduce a fast, low-cost method for creating diverse
ensembles of neural networks without needing to train multiple models from
scratch. We do this by first training a single parent network. We then create
child networks by cloning the parent and dramatically pruning the parameters of
each child to create an ensemble of members with unique and diverse topologies.
We then briefly train each child network for a small number of epochs, which
now converge significantly faster when compared to training from scratch. We
explore various ways to maximize diversity in the child networks, including the
use of anti-random pruning and one-cycle tuning. This diversity enables "Prune
and Tune" ensembles to achieve results that are competitive with traditional
ensembles at a fraction of the training cost. We benchmark our approach against
state of the art low-cost ensemble methods and display marked improvement in
both accuracy and uncertainty estimation on CIFAR-10 and CIFAR-100.

本文提出一种名为 “Prune and Tune” 的方法，通过剪枝单个神经网络的参数创建多个包含不同拓扑结构的成员作为集成，最终使得训练集成成员的代价明显降低同时准确率不降反升，主要应用于 CIFAR-10 和 CIFAR-100 数据集。