Large-scale pre-trained models have been remarkably successful in resolving
downstream tasks. Nonetheless, deploying these models on low-capability devices
still requires an effective approach, such as model pruning. However, pruning
the model from scratch can pose a practical challenge given the limited
resources of each downstream task or device. To tackle this issue, we present a
scalable one-shot pruning method that leverages pruned knowledge of similar
tasks to extract a sub-network from the pre-trained model for a new task.
Specifically, we create a score mask using the pruned models of similar tasks
to identify task-specific filters/nodes in the pre-trained model for the new
task. Based on this mask, we conduct a single round of pruning to extract a
suitably-sized sub-network that can quickly adapt to the new task with only a
few training iterations. Our experimental analysis demonstrates the
effectiveness of the proposed method on the convolutional neural networks
(CNNs) and vision transformers (ViT) with various datasets. The proposed method
consistently outperforms popular pruning baseline methods in terms of accuracy
and efficiency when dealing with diverse downstream tasks with different memory
constraints.

基于大规模预训练模型，提出了一种适用于低能力设备的可伸缩单次剪枝方法，利用类似任务的剪枝知识从预训练模型中提取一个子网络来适应新任务，实验证明该方法在处理具有不同内存限制的多样化下游任务时，在准确性和效率方面始终优于流行的剪枝基准方法。