Very large-scale Deep Neural Networks (DNNs) have achieved remarkable
successes in a large variety of computer vision tasks. However, the high
computation intensity of DNNs makes it challenging to deploy these models on
resource-limited systems. Some studies used low-rank approaches that
approximate the filters by low-rank basis to accelerate the testing. Those
works directly decomposed the pre-trained DNNs by Low-Rank Approximations
(LRA). How to train DNNs toward lower-rank space for more efficient DNNs,
however, remains as an open area. To solve the issue, in this work, we propose
Force Regularization, which uses attractive forces to enforce filters so as to
coordinate more weight information into lower-rank space. We mathematically and
empirically verify that after applying our technique, standard LRA methods can
reconstruct filters using much lower basis and thus result in faster DNNs. The
effectiveness of our approach is comprehensively evaluated in ResNets, AlexNet,
and GoogLeNet. In AlexNet, for example, Force Regularization gains 2x speedup
on modern GPU without accuracy loss and 4.05x speedup on CPU by paying small
accuracy degradation. Moreover, Force Regularization better initializes the
low-rank DNNs such that the fine-tuning can converge faster toward higher
accuracy. The obtained lower-rank DNNs can be further sparsified, proving that
Force Regularization can be integrated with state-of-the-art sparsity-based
acceleration methods. Source code is available in
this https URL

本文提出 Force Regularization 方法，利用引力作用来协调更多的权重信息到低秩空间中，从而使得标准的低秩逼近方法可以使用更少的基础滤波器进行滤波，加快 DNN 速度，该方法在 ResNet，AlexNet 和 GoogLeNet 上进行了综合评估，在现代 GPU 上获得了 2 倍的加速效果而没有损失精度，获得了 4.05 倍的加速比，可以与最先进的稀疏加速方法集成。