State-of-the-art convolutional neural networks (CNNs) used in vision
applications have large models with numerous weights. Training these models is
very compute- and memory-resource intensive. Much research has been done on
pruning or compressing these models to reduce the cost of inference, but little
work has addressed the costs of training. We focus precisely on accelerating
training. We propose PruneTrain, a cost-efficient mechanism that gradually
reduces the training cost during training. PruneTrain uses a structured
group-lasso regularization approach that drives the training optimization
toward both high accuracy and small weight values. Small weights can then be
periodically removed by reconfiguring the network model to a smaller one. By
using a structured-pruning approach and additional reconfiguration techniques
we introduce, the pruned model can still be efficiently processed on a GPU
accelerator. Overall, PruneTrain achieves a reduction of 39% in the end-to-end
training time of ResNet50 for ImageNet by reducing computation cost by 40% in
FLOPs, memory accesses by 37% for memory bandwidth bound layers, and the
inter-accelerator communication by 55%.

我们提出了 PruneTrain，这是一种成本高效的机制，可逐步减少训练期间的计算成本，通过使用结构化组套骨骼正则化方法和其他重新配置技术，可以在 GPU 加速器上高效处理缩小了的卷积神经网络模型，从而实现减少计算成本 40％和训练时间 39％的效果。