Neural networks have achieved tremendous success in a large variety of
applications. However, their memory footprint and computational demand can
render them impractical in application settings with limited hardware or energy
resources. In this work, we propose a novel algorithm to find efficient
low-rank subnetworks. Remarkably, these subnetworks are determined and adapted
already during the training phase and the overall time and memory resources
required by both training and evaluating them are significantly reduced. The
main idea is to restrict the weight matrices to a low-rank manifold and to
update the low-rank factors rather than the full matrix during training. To
derive training updates that are restricted to the prescribed manifold, we
employ techniques from dynamic model order reduction for matrix differential
equations. This allows us to provide approximation, stability, and descent
guarantees. Moreover, our method automatically and dynamically adapts the ranks
during training to achieve the desired approximation accuracy. The efficiency
of the proposed method is demonstrated through a variety of numerical
experiments on fully-connected and convolutional networks.

本文提出了一种利用低秩子网提高神经网络效率的算法，该算法在训练期间已经 determined and adapted the subnetworks，从而大幅减少了训练和评估所需的时间和内存资源，并通过数值实验验证了该算法的高效性。