This paper explores the feasibility of finding an optimal sub-model from a
vision transformer and introduces a pure vision transformer slimming (ViT-Slim)
framework. It can search a sub-structure from the original model end-to-end
across multiple dimensions, including the input tokens, MHSA and MLP modules
with state-of-the-art performance. Our method is based on a learnable and
unified $\ell_1$ sparsity constraint with pre-defined factors to reflect the
global importance in the continuous searching space of different dimensions.
The searching process is highly efficient through a single-shot training
scheme. For instance, on DeiT-S, ViT-Slim only takes ~43 GPU hours for the
searching process, and the searched structure is flexible with diverse
dimensionalities in different modules. Then, a budget threshold is employed
according to the requirements of accuracy-FLOPs trade-off on running devices,
and a re-training process is performed to obtain the final model. The extensive
experiments show that our ViT-Slim can compress up to 40% of parameters and 40%
FLOPs on various vision transformers while increasing the accuracy by ~0.6% on
ImageNet. We also demonstrate the advantage of our searched models on several
downstream datasets. Our code is available at
this https URL.

本文介绍了一种名为 ViT-Slim 的视觉转换器压缩框架，基于可学习和统一的 l1 稀疏性约束搜索多个维度的子结构，通过预算阈值得到压缩后的模型，试验结果表明，ViT-Slim 可以在保持较高图像分类准确率的同时，将参数和 FLOP 压缩至 40%。