As Vision Transformers (ViTs) increasingly set new benchmarks in computer
vision, their practical deployment on inference engines is often hindered by
their significant memory bandwidth and (on-chip) memory footprint requirements.
This paper addresses this memory limitation by introducing an activation-aware
model compression methodology that uses selective low-rank weight tensor
approximations of different layers to reduce the parameter count of ViTs. The
key idea is to decompose the weight tensors into a sum of two
parameter-efficient tensors while minimizing the error between the product of
the input activations with the original weight tensor and the product of the
input activations with the approximate tensor sum. This approximation is
further refined by adopting an efficient layer-wise error compensation
technique that uses the gradient of the layer's output loss. The combination of
these techniques achieves excellent results while it avoids being trapped in a
shallow local minimum early in the optimization process and strikes a good
balance between the model compression and output accuracy. Notably, the
presented method significantly reduces the parameter count of DeiT-B by 60%
with less than 1% accuracy drop on the ImageNet dataset, overcoming the usual
accuracy degradation seen in low-rank approximations. In addition to this, the
presented compression technique can compress large DeiT/ViT models to have
about the same model size as smaller DeiT/ViT variants while yielding up to
1.8% accuracy gain. These results highlight the efficacy of our approach,
presenting a viable solution for embedding ViTs in memory-constrained
environments without compromising their performance.

本文提出了一种基于激活感知的模型压缩方法，通过使用不同层的低秩张量逼近来减少 Vision Transformers（ViTs）的参数数量，克服了内存限制，实现在内存受限环境中嵌入 ViTs 而不影响其性能。