Compressing DNNs is important for the real-world applications operating on
resource-constrained devices. However, we typically observe drastic performance
deterioration when changing model size after training is completed. Therefore,
retraining is required to resume the performance of the compressed models
suitable for different devices. In this paper, we propose Decomposable-Net (the
network decomposable in any size), which allows flexible changes to model size
without retraining. We decompose weight matrices in the DNNs via singular value
decomposition and adjust ranks according to the target model size. Unlike the
existing low-rank compression methods that specialize the model to a fixed
size, we propose a novel backpropagation scheme that jointly minimizes losses
for both of full- and low-rank networks. This enables not only to maintain the
performance of a full-rank network {\it without retraining} but also to improve
low-rank networks in multiple sizes. Additionally, we introduce a simple
criterion for rank selection that effectively suppresses approximation error.
In experiments on the ImageNet classification task, Decomposable-Net yields
superior accuracy in a wide range of model sizes. In particular,
Decomposable-Net achieves the top-1 accuracy of $73.2\%$ with $0.27\times$MACs
with ResNet-50, compared to Tucker decomposition ($67.4\% / 0.30\times$),
Trained Rank Pruning ($70.6\% / 0.28\times$), and universally slimmable
networks ($71.4\% / 0.26\times$).

本文介绍了一种名为 Decomposable-Net 的深度神经网络压缩方法，通过奇异值分解和调整矩阵秩，允许灵活改变模型大小，而无需进行微调，能够在多种模型大小下提高模型性能。