Compressing a predefined deep neural network (DNN) into a compact sub-network
with competitive performance is crucial in the efficient machine learning
realm. This topic spans various techniques, from structured pruning to neural
architecture search, encompassing both pruning and erasing operators
perspectives. Despite advancements, existing methods suffers from complex,
multi-stage processes that demand substantial engineering and domain knowledge,
limiting their broader applications. We introduce the third-generation
Only-Train-Once (OTOv3), which first automatically trains and compresses a
general DNN through pruning and erasing operations, creating a compact and
competitive sub-network without the need of fine-tuning. OTOv3 simplifies and
automates the training and compression process, minimizes the engineering
efforts required from users. It offers key technological advancements: (i)
automatic search space construction for general DNNs based on dependency graph
analysis; (ii) Dual Half-Space Projected Gradient (DHSPG) and its enhanced
version with hierarchical search (H2SPG) to reliably solve (hierarchical)
structured sparsity problems and ensure sub-network validity; and (iii)
automated sub-network construction using solutions from DHSPG/H2SPG and
dependency graphs. Our empirical results demonstrate the efficacy of OTOv3
across various benchmarks in structured pruning and neural architecture search.
OTOv3 produces sub-networks that match or exceed the state-of-the-arts. The
source code will be available at this https URL

通过压缩和擦除操作，自动训练和压缩一般 DNN，创建紧凑且有竞争力的子网络，无需微调，简化和自动化训练和压缩过程，减少用户的工程工作。