To accelerate DNNs inference, low-rank approximation has been widely adopted
because of its solid theoretical rationale and efficient implementations.
Several previous works attempted to directly approximate a pre-trained model by
low-rank decomposition; however, small approximation errors in parameters can
ripple over a large prediction loss. Apparently, it is not optimal to separate
low-rank approximation from training. Unlike previous works, this paper
integrates low rank approximation and regularization into the training process.
We propose Trained Rank Pruning (TRP), which alternates between low rank
approximation and training. TRP maintains the capacity of the original network
while imposing low-rank constraints during training. A nuclear regularization
optimized by stochastic sub-gradient descent is utilized to further promote low
rank in TRP. Networks trained with TRP has a low-rank structure in nature, and
is approximated with negligible performance loss, thus eliminating fine-tuning
after low rank approximation. The proposed method is comprehensively evaluated
on CIFAR-10 and ImageNet, outperforming previous compression counterparts using
low rank approximation. Our code is available at:
this https URL

文章提出了一种名为 TRP 的方法，该方法可以将低秩逼近和正则化融入神经网络模型的训练过程中，通过使用随机次梯度下降优化核范数来促进 TRP 中的低秩，从而消除低秩逼近后的微调，并在 CIFAR-10 和 ImageNet 上显着优于以前使用低秩逼近的对比方法。

高效深度神经网络的训练等级裁剪

Trained Rank Pruning for Efficient Deep Neural Networks

The performance of Deep Neural Networks (DNNs) keeps elevating in recent
years with increasing network depth and width. To enable DNNs on edge devices
like mobile phones, researchers proposed several network compression methods
including pruning, quantization and factorization. Among the
factorization-based approaches, low-rank approximation has been widely adopted
because of its solid theoretical rationale and efficient implementations.
Several previous works attempted to directly approximate a pre-trained model by
low-rank decomposition; however, small approximation errors in parameters can
ripple a large prediction loss. As a result, performance usually drops
significantly and a sophisticated fine-tuning is required to recover accuracy.
We argue that it is not optimal to separate low-rank approximation from
training. Unlike previous works, this paper integrates low rank approximation
and regularization into the training. We propose Trained Rank Pruning (TRP),
which iterates low rank approximation and training. TRP maintains the capacity
of original network while imposes low-rank constraints during training. A
stochastic sub-gradient descent optimized nuclear regularization is utilized to
further encourage low rank in TRP. The TRP trained network has low-rank
structure in nature, and can be approximated with negligible performance loss,
eliminating fine-tuning after low rank approximation. The methods are
comprehensively evaluated on CIFAR-10 and ImageNet, outperforming previous
compression methods using low rank approximation. Code is available:
this https URL

本文提出了一种名为 TRP 的低秩逼近和正则化相结合的训练方法，用于在边缘设备上进行深度神经网络的图像分类任务，并在 CIFAR-10 和 ImageNet 数据集上进行了广泛评估，结果表明相比于之前的低秩逼近压缩方法取得了更好的性能。