Convolutional Neural Networks (CNNs) are hard to deploy on edge devices due
to its high computation and storage complexities. As a common practice for
model compression, network pruning consists of two major categories:
unstructured and structured pruning, where unstructured pruning constantly
performs better. However, unstructured pruning presents a structured pattern at
high pruning rates, which limits its performance. To this end, we propose a
Rank-based PruninG (RPG) method to maintain the ranks of sparse weights in an
adversarial manner. In each step, we minimize the low-rank approximation error
for the weight matrices using singular value decomposition, and maximize their
distance by pushing the weight matrices away from its low rank approximation.
This rank-based optimization objective guides sparse weights towards a
high-rank topology. The proposed method is conducted in a gradual pruning
fashion to stabilize the change of rank during training. Experimental results
on various datasets and different tasks demonstrate the effectiveness of our
algorithm in high sparsity. The proposed RPG outperforms the state-of-the-art
performance by 1.13% top-1 accuracy on ImageNet in ResNet-50 with 98% sparsity.
The codes are available at
this https URL and
this https URL

卷积神经网络在边缘设备上部署困难，提出了一种基于排名的修剪方法（Rank-based PruninG），通过稀疏权重的高秩拓扑结构来实现高稀疏度，该方法在各种数据集和不同任务上验证了其有效性。

通过对抗性权重修剪实现更高的排名

Towards Higher Ranks via Adversarial Weight Pruning

Obtaining versions of deep neural networks that are both highly-accurate and
highly-sparse is one of the main challenges in the area of model compression,
and several high-performance pruning techniques have been investigated by the
community. Yet, much less is known about the interaction between sparsity and
the standard stochastic optimization techniques used for training sparse
networks, and most existing work uses standard dense schedules and
hyperparameters for training sparse networks. In this work, we examine the
impact of high sparsity on model training using the standard computer vision
and natural language processing sparsity benchmarks. We begin by showing that
using standard dense training recipes for sparse training is suboptimal, and
results in under-training. We provide new approaches for mitigating this issue
for both sparse pre-training of vision models (e.g. ResNet50/ImageNet) and
sparse fine-tuning of language models (e.g. BERT/GLUE), achieving
state-of-the-art results in both settings in the high-sparsity regime, and
providing detailed analyses for the difficulty of sparse training in both
scenarios. Our work sets a new threshold in terms of the accuracies that can be
achieved under high sparsity, and should inspire further research into
improving sparse model training, to reach higher accuracies under high
sparsity, but also to do so efficiently.

使用标准的计算机视觉和自然语言处理稀疏基准测试，探讨高稀疏性对模型训练的影响，提供了一种解决有关稀疏训练困难的新方法，并在高稀疏性环境下实现了在视觉模型和语言模型上最先进的结果。