Deep neural network models have a complex architecture and are overparameterized. The number of parameters is more than the whole dataset, which is highly resource-consuming. This complicates their application and limits its usage on different devices. Reduction in the number of network parameters helps to reduce the size of the model, but at the same time, thoughtlessly applied, can lead to a deterioration in the quality of the network. One way to reduce the number of model parameters is matrix decomposition, where a matrix is represented as a product of smaller matrices. In this paper, we propose a new way of applying the matrix decomposition with respect to the weights of convolutional layers. The essence of the method is to train not all convolutions, but only the subset of convolutions (basis convolutions), and represent the rest as linear combinations of the basis ones. Experiments on models from the ResNet family and the CIFAR-10 dataset demonstrate that basis convolutions can not only reduce the size of the model but also accelerate the forward and backward passes of the network. Another contribution of this work is that we propose a fast method for selecting a subset of network layers in which the use of matrix decomposition does not degrade the quality of the final model.

本研究针对深度神经网络模型中参数过多导致的资源消耗问题，提出了一种新颖的矩阵分解方法，专注于训练卷积层的部分基础卷积，而将其余卷积表示为基础卷积的线性组合。实验结果表明，该方法不仅能有效减少模型大小，还能加速网络的前向和反向传播，并且提出了快速选择不影响模型质量的网络层子集的方法。

层特定优化：基于敏感性的卷积层基础搜索