Neural network pruning is a key technique towards engineering large yet scalable, interpretable, and generalizable models. Prior work on the subject has developed largely along two orthogonal directions: (1) differentiable pruning for efficiently and accurately scoring the importance of parameters, and (2) combinatorial optimization for efficiently searching over the space of sparse models. We unite the two approaches, both theoretically and empirically, to produce a coherent framework for structured neural network pruning in which differentiable pruning guides combinatorial optimization algorithms to select the most important sparse set of parameters. Theoretically, we show how many existing differentiable pruning techniques can be understood as nonconvex regularization for group sparse optimization, and prove that for a wide class of nonconvex regularizers, the global optimum is unique, group-sparse, and provably yields an approximate solution to a sparse convex optimization problem. The resulting algorithm that we propose, SequentialAttention++, advances the state of the art in large-scale neural network block-wise pruning tasks on the ImageNet and Criteo datasets.

神经网络剪枝是一种重要的技术，能够构建大规模且可扩展、可解释和可推广模型。本文提出了一个结构化神经网络剪枝的方法，将可微剪枝指导组合优化算法选择最重要的稀疏参数集，从而在ImageNet和Criteo数据集上取得了大规模神经网络块级剪枝任务的最新成果。

基于顺序注意力的块稀疏化: 可微剪枝结合组合优化