Factorized layers--operations parameterized by products of two or more
matrices--occur in a variety of deep learning contexts, including compressed
model training, certain types of knowledge distillation, and multi-head
self-attention architectures. We study how to initialize and regularize deep
nets containing such layers, examining two simple, understudied schemes,
spectral initialization and Frobenius decay, for improving their performance.
The guiding insight is to design optimization routines for these networks that
are as close as possible to that of their well-tuned, non-decomposed
counterparts; we back this intuition with an analysis of how the initialization
and regularization schemes impact training with gradient descent, drawing on
modern attempts to understand the interplay of weight-decay and
batch-normalization. Empirically, we highlight the benefits of spectral
initialization and Frobenius decay across a variety of settings. In model
compression, we show that they enable low-rank methods to significantly
outperform both unstructured sparsity and tensor methods on the task of
training low-memory residual networks; analogs of the schemes also improve the
performance of tensor decomposition techniques. For knowledge distillation,
Frobenius decay enables a simple, overcomplete baseline that yields a compact
model from over-parameterized training without requiring retraining with or
pruning a teacher network. Finally, we show how both schemes applied to
multi-head attention lead to improved performance on both translation and
unsupervised pre-training.

研究深度学习中出现的因子化层，并探讨如何通过谱初始化和 Frobenius 衰减来改善它们的性能，并在模型压缩和知识蒸馏领域进行了实验。