To accelerate DNNs inference, low-rank approximation has been widely adopted
because of its solid theoretical rationale and efficient implementations.
Several previous works attempted to directly approximate a pre-trained model by
low-rank decomposition; however, small approximation errors in parameters can
ripple over a large prediction loss. Apparently, it is not optimal to separate
low-rank approximation from training. Unlike previous works, this paper
integrates low rank approximation and regularization into the training process.
We propose Trained Rank Pruning (TRP), which alternates between low rank
approximation and training. TRP maintains the capacity of the original network
while imposing low-rank constraints during training. A nuclear regularization
optimized by stochastic sub-gradient descent is utilized to further promote low
rank in TRP. Networks trained with TRP has a low-rank structure in nature, and
is approximated with negligible performance loss, thus eliminating fine-tuning
after low rank approximation. The proposed method is comprehensively evaluated
on CIFAR-10 and ImageNet, outperforming previous compression counterparts using
low rank approximation. Our code is available at:
this https URL

文章提出了一种名为 TRP 的方法，该方法可以将低秩逼近和正则化融入神经网络模型的训练过程中，通过使用随机次梯度下降优化核范数来促进 TRP 中的低秩，从而消除低秩逼近后的微调，并在 CIFAR-10 和 ImageNet 上显着优于以前使用低秩逼近的对比方法。