Compressing neural networks is a key step when deploying models for real-time
or embedded applications. Factorizing the model's matrices using low-rank
approximations is a promising method for achieving compression. While it is
possible to set the rank before training, this approach is neither flexible nor
optimal. In this work, we propose a post-training rank-selection method called
Rank-Tuning that selects a different rank for each matrix. Used in combination
with training adaptations, our method achieves high compression rates with no
or little performance degradation. Our numerical experiments on signal
processing tasks show that we can compress recurrent neural networks up to 14x
with at most 1.4% relative performance reduction.

基于训练后的秩选择方法 Rank-Tuning，结合训练适应性，我们的方法能够实现高压缩率而无性能损失或性能损失较小，数值实验结果表明，我们可以将循环神经网络压缩至最多 14 倍，最多相对性能降低 1.4%。

利用矩阵分解对循环神经网络进行压缩

Compression of Recurrent Neural Networks using Matrix Factorization

Compressing deep neural networks while maintaining accuracy is important when
we want to deploy large, powerful models in production and/or edge devices. One
common technique used to achieve this goal is knowledge distillation.
Typically, the output of a static pre-defined teacher (a large base network) is
used as soft labels to train and transfer information to a student (or smaller)
network. In this paper, we introduce Adjoined Networks, or AN, a learning
paradigm that trains both the original base network and the smaller compressed
network together. In our training approach, the parameters of the smaller
network are shared across both the base and the compressed networks. Using our
training paradigm, we can simultaneously compress (the student network) and
regularize (the teacher network) any architecture. In this paper, we focus on
popular CNN-based architectures used for computer vision tasks. We conduct an
extensive experimental evaluation of our training paradigm on various
large-scale datasets. Using ResNet-50 as the base network, AN achieves 71.8%
top-1 accuracy with only 1.8M parameters and 1.6 GFLOPs on the ImageNet
data-set. We further propose Differentiable Adjoined Networks (DAN), a training
paradigm that augments AN by using neural architecture search to jointly learn
both the width and the weights for each layer of the smaller network. DAN
achieves ResNet-50 level accuracy on ImageNet with $3.8\times$ fewer parameters
and $2.2\times$ fewer FLOPs.

介绍了 Adjoined Networks，一种新的神经网络压缩方法，采用同时训练基础网络和压缩网络的策略，通过压缩并规范化模型，使得模型在保持精度的同时，也具有较小的存储和计算需求。此外，提出了 Differentiable Adjoined Networks，利用神经网络搜索技术，进一步提升了模型的压缩效果。