Current techniques for deep neural network (DNN) pruning often involve
intricate multi-step processes that require domain-specific expertise, making
their widespread adoption challenging. To address the limitation, the
Only-Train-Once (OTO) and OTOv2 are proposed to eliminate the need for
additional fine-tuning steps by directly training and compressing a general DNN
from scratch. Nevertheless, the static design of optimizers (in OTO) can lead
to convergence issues of local optima. In this paper, we proposed the
Auto-Train-Once (ATO), an innovative network pruning algorithm designed to
automatically reduce the computational and storage costs of DNNs. During the
model training phase, our approach not only trains the target model but also
leverages a controller network as an architecture generator to guide the
learning of target model weights. Furthermore, we developed a novel stochastic
gradient algorithm that enhances the coordination between model training and
controller network training, thereby improving pruning performance. We provide
a comprehensive convergence analysis as well as extensive experiments, and the
results show that our approach achieves state-of-the-art performance across
various model architectures (including ResNet18, ResNet34, ResNet50, ResNet56,
and MobileNetv2) on standard benchmark datasets (CIFAR-10, CIFAR-100, and
ImageNet).

我们提出了 Auto-Train-Once（ATO）算法，旨在自动减少深度神经网络的计算和存储成本，通过训练目标模型同时利用控制器网络作为架构生成器来指导目标模型权重的学习，并利用新颖的随机梯度算法增强模型训练和控制器网络训练之间的协调，从而提高剪枝性能。在本文中，我们提供了全面的收敛分析和大量实验，结果表明我们的方法在各种模型架构（包括 ResNet18、ResNet34、ResNet50、ResNet56 和 MobileNetv2）以及标准基准数据集（CIFAR-10、CIFAR-100 和 ImageNet）上实现了最先进的性能。