Modern deep neural networks (DNNs) often require high memory consumption and
large computational loads. In order to deploy DNN algorithms efficiently on
edge or mobile devices, a series of DNN compression algorithms have been
explored, including factorization methods. Factorization methods approximate
the weight matrix of a DNN layer with the multiplication of two or multiple
low-rank matrices. However, it is hard to measure the ranks of DNN layers
during the training process. Previous works mainly induce low-rank through
implicit approximations or via costly singular value decomposition (SVD)
process on every training step. The former approach usually induces a high
accuracy loss while the latter has a low efficiency. In this work, we propose
SVD training, the first method to explicitly achieve low-rank DNNs during
training without applying SVD on every step. SVD training first decomposes each
layer into the form of its full-rank SVD, then performs training directly on
the decomposed weights. We add orthogonality regularization to the singular
vectors, which ensure the valid form of SVD and avoid gradient
vanishing/exploding. Low-rank is encouraged by applying sparsity-inducing
regularizers on the singular values of each layer. Singular value pruning is
applied at the end to explicitly reach a low-rank model. We empirically show
that SVD training can significantly reduce the rank of DNN layers and achieve
higher reduction on computation load under the same accuracy, comparing to not
only previous factorization methods but also state-of-the-art filter pruning
methods.

本研究提出了一种名为 SVD training 的神经网络压缩方法，通过正交性正则化和奇异值剪枝等技术，可以在训练过程中显式地实现降低矩阵秩的目标，从而更有效地减少 DNN 算法在低性能设备上的计算负担。