Vision Transformers (ViT) have recently demonstrated success across a myriad
of computer vision tasks. However, their elevated computational demands pose
significant challenges for real-world deployment. While low-rank approximation
stands out as a renowned method to reduce computational loads, efficiently
automating the target rank selection in ViT remains a challenge. Drawing from
the notable similarity and alignment between the processes of rank selection
and One-Shot NAS, we introduce FLORA, an end-to-end automatic framework based
on NAS. To overcome the design challenge of supernet posed by vast search
space, FLORA employs a low-rank aware candidate filtering strategy. This method
adeptly identifies and eliminates underperforming candidates, effectively
alleviating potential undertraining and interference among subnetworks. To
further enhance the quality of low-rank supernets, we design a low-rank
specific training paradigm. First, we propose weight inheritance to construct
supernet and enable gradient sharing among low-rank modules. Secondly, we adopt
low-rank aware sampling to strategically allocate training resources, taking
into account inherited information from pre-trained models. Empirical results
underscore FLORA's efficacy. With our method, a more fine-grained rank
configuration can be generated automatically and yield up to 33% extra FLOPs
reduction compared to a simple uniform configuration. More specific,
FLORA-DeiT-B/FLORA-Swin-B can save up to 55%/42% FLOPs almost without
performance degradtion. Importantly, FLORA boasts both versatility and
orthogonality, offering an extra 21%-26% FLOPs reduction when integrated with
leading compression techniques or compact hybrid structures. Our code is
publicly available at this https URL

通过引入基于 NAS 的自动化框架 FLORA，本论文提出了一种用于降低计算负载的低秩逼近方法，通过低秩感知的候选过滤策略和低秩特定训练方法，实现了更精细的低秩配置，相比简单均匀配置可额外减少 33% 的操作次数，同时还展示了与压缩技术和紧凑混合结构整合可达到更高的 FLOPs 减少率。