Recent research has shown that training low-rank neural networks can
effectively reduce the total number of trainable parameters without sacrificing
predictive accuracy, resulting in end-to-end speedups. However, low-rank model
training necessitates adjusting several additional factorization
hyperparameters, such as the rank of the factorization at each layer. In this
paper, we tackle this challenge by introducing Cuttlefish, an automated
low-rank training approach that eliminates the need for tuning factorization
hyperparameters. Cuttlefish leverages the observation that after a few epochs
of full-rank training, the stable rank (i.e., an approximation of the true
rank) of each layer stabilizes at a constant value. Cuttlefish switches from
full-rank to low-rank training once the stable ranks of all layers have
converged, setting the dimension of each factorization to its corresponding
stable rank. Our results show that Cuttlefish generates models up to 5.6 times
smaller than full-rank models, and attains up to a 1.2 times faster end-to-end
training process while preserving comparable accuracy. Moreover, Cuttlefish
outperforms state-of-the-art low-rank model training methods and other
prominent baselines. The source code for our implementation can be found at:
this https URL

提出了一种名为 Cuttlefish 的自动低秩训练方法，它通过观察在完整秩训练的几个时期后，每层的稳定秩（即真实秩的近似值）稳定为一个常数值，从而消除了调整低秩模型训练中的多个超参数的需求。Cuttlefish 在所有层的稳定秩收敛后从完整秩训练切换到低秩训练，并将每个因子化的维度设置为其相应的稳定秩。结果表明，Cuttlefish 生成的模型比完整秩模型小多达 5.6 倍，并达到了 1.2 倍的更快的端到端训练过程，同时保持相当的准确性，在低秩模型训练和其他著名基线模型中表现出色。