Deep Neural Networks (DNNs) have achieved remarkable success in addressing
many previously unsolvable tasks. However, the storage and computational
requirements associated with DNNs pose a challenge for deploying these trained
models on resource-limited devices. Therefore, a plethora of compression and
pruning techniques have been proposed in recent years. Low-rank decomposition
techniques are among the approaches most utilized to address this problem.
Compared to post-training compression, compression-promoted training is still
under-explored. In this paper, we present a theoretically-justified novel
approach, termed Low-Rank Induced Training (LoRITa), that promotes low-rankness
through the composition of linear layers and compresses by using singular value
truncation. This is achieved without the need to change the structure at
inference time or require constrained and/or additional optimization, other
than the standard weight decay regularization. Moreover, LoRITa eliminates the
need to (i) initialize with pre-trained models and (ii) specify rank selection
prior to training. Our experimental results (i) demonstrate the effectiveness
of our approach using MNIST on Fully Connected Networks, CIFAR10 on Vision
Transformers, and CIFAR10/100 on Convolutional Neural Networks, and (ii)
illustrate that we achieve either competitive or SOTA results when compared to
leading structured pruning methods in terms of FLOPs and parameters drop.

本研究论文介绍了一种名为低秩引导训练（LoRITa）的压缩技术，通过组合线性层和使用奇异值截断来促进低秩性，并且在推理时无需改变结构或进行额外的优化，通过实验证明了其有效性，并与其他主流结构剪枝方法相比在 FLOPs 和参数减少方面取得了竞争性或 SOTA 结果。