Vision transformers (ViTs) have recently received explosive popularity, but
their enormous model sizes and training costs remain daunting. Conventional
post-training pruning often incurs higher training budgets. In contrast, this
paper aims to trim down both the training memory overhead and the inference
complexity, without sacrificing the achievable accuracy. We carry out the
first-of-its-kind comprehensive exploration, on taking a unified approach of
integrating sparsity in ViTs "from end to end". Specifically, instead of
training full ViTs, we dynamically extract and train sparse subnetworks, while
sticking to a fixed small parameter budget. Our approach jointly optimizes
model parameters and explores connectivity throughout training, ending up with
one sparse network as the final output. The approach is seamlessly extended
from unstructured to structured sparsity, the latter by considering to guide
the prune-and-grow of self-attention heads inside ViTs. We further co-explore
data and architecture sparsity for additional efficiency gains by plugging in a
novel learnable token selector to adaptively determine the currently most vital
patches. Extensive results on ImageNet with diverse ViT backbones validate the
effectiveness of our proposals which obtain significantly reduced computational
cost and almost unimpaired generalization. Perhaps most surprisingly, we find
that the proposed sparse (co-)training can sometimes improve the ViT accuracy
rather than compromising it, making sparsity a tantalizing "free lunch". For
example, our sparsified DeiT-Small at (5%, 50%) sparsity for (data,
architecture), improves 0.28% top-1 accuracy, and meanwhile enjoys 49.32% FLOPs
and 4.40% running time savings. Our codes are available at
this https URL

本文旨在从减小训练存储开销和推理复杂度的角度，提出一种先将 Vision transformers 稀疏化，然后再训练的方法，从而实现一定的加速效果并保持较高的精度。