The low displacement rank (LDR) framework for structured matrices represents
a matrix through two displacement operators and a low-rank residual. Existing
use of LDR matrices in deep learning has applied fixed displacement operators
encoding forms of shift invariance akin to convolutions. We introduce a class
of LDR matrices with more general displacement operators, and explicitly learn
over both the operators and the low-rank component. This class generalizes
several previous constructions while preserving compression and efficient
computation. We prove bounds on the VC dimension of multi-layer neural networks
with structured weight matrices and show empirically that our compact
parameterization can reduce the sample complexity of learning. When replacing
weight layers in fully-connected, convolutional, and recurrent neural networks
for image classification and language modeling tasks, our new classes exceed
the accuracy of existing compression approaches, and on some tasks also
outperform general unstructured layers while using more than 20x fewer
parameters.

通过更一般化的位移操作，显式地学习两个位移算子和低秩分量，设计出一种新的 LDR 矩阵，并证明其在多层神经网络的 VC 维度上具有较好的界限，有效地降低模型复杂度。实验结果表明，该设计可以优于现有的压缩方法，并使用的参数数量比一般非结构化网络少 20 倍以上。

低位移秩学习压缩变换

Learning Compressed Transforms with Low Displacement Rank

We consider the task of building compact deep learning pipelines suitable for
deployment on storage and power constrained mobile devices. We propose a
unified framework to learn a broad family of structured parameter matrices that
are characterized by the notion of low displacement rank. Our structured
transforms admit fast function and gradient evaluation, and span a rich range
of parameter sharing configurations whose statistical modeling capacity can be
explicitly tuned along a continuum from structured to unstructured.
Experimental results show that these transforms can significantly accelerate
inference and forward/backward passes during training, and offer superior
accuracy-compactness-speed tradeoffs in comparison to a number of existing
techniques. In keyword spotting applications in mobile speech recognition, our
methods are much more effective than standard linear low-rank bottleneck layers
and nearly retain the performance of state of the art models, while providing
more than 3.5-fold compression.

提出了一种基于低位移秩概念的结构化变换方法，能快速优化深度学习模型在储存和功耗受限移动设备上的部署，通过参数共享的各种配置实现结构化到非结构化的统计建模，并在关键词检测应用中显著提高推理速度和轻量化程度，表现优于目前的技术。