This paper investigates efficient deep neural networks (DNNs) to replace
dense unstructured weight matrices with structured ones that possess desired
properties. The challenge arises because the optimal weight matrix structure in
popular neural network models is obscure in most cases and may vary from layer
to layer even in the same network. Prior structured matrices proposed for
efficient DNNs were mostly hand-crafted without a generalized framework to
systematically learn them. To address this issue, we propose a generalized and
differentiable framework to learn efficient structures of weight matrices by
gradient descent. We first define a new class of structured matrices that
covers a wide range of structured matrices in the literature by adjusting the
structural parameters. Then, the frequency-domain differentiable
parameterization scheme based on the Gaussian-Dirichlet kernel is adopted to
learn the structural parameters by proximal gradient descent. Finally, we
introduce an effective initialization method for the proposed scheme. Our
method learns efficient DNNs with structured matrices, achieving lower
complexity and/or higher performance than prior approaches that employ
low-rank, block-sparse, or block-low-rank matrices.

该论文研究了如何通过梯度下降学习结构化权重矩阵的一般化和可微分的框架，以取代常见神经网络模型中不透明的、层与层之间可能不同的密集矩阵，从而构建高性能且复杂度较低的高效深度神经网络。