Large language models (LLMs) have shown impressive capabilities across
various tasks. However, training LLMs from scratch requires significant
computational power and extensive memory capacity. Recent studies have explored
low-rank structures on weights for efficient fine-tuning in terms of parameters
and memory, either through low-rank adaptation or factorization. While
effective for fine-tuning, low-rank structures are generally less suitable for
pretraining because they restrict parameters to a low-dimensional subspace. In
this work, we propose to parameterize the weights as a sum of low-rank and
sparse matrices for pretraining, which we call SLTrain. The low-rank component
is learned via matrix factorization, while for the sparse component, we employ
a simple strategy of uniformly selecting the sparsity support at random and
learning only the non-zero entries with the fixed support. While being simple,
the random fixed-support sparse learning strategy significantly enhances
pretraining when combined with low-rank learning. Our results show that SLTrain
adds minimal extra parameters and memory costs compared to pretraining with
low-rank parameterization, yet achieves substantially better performance, which
is comparable to full-rank training. Remarkably, when combined with
quantization and per-layer updates, SLTrain can reduce memory requirements by
up to 73% when pretraining the LLaMA 7B model.

通过在预训练中将权重参数化为低秩和稀疏矩阵之和，我们提出了一种称为 SLTrain 的方法，它通过矩阵分解学习低秩部分，然后使用随机固定支撑稀疏学习策略学习非零条目，这种策略显著提高了预训练性能。与低秩参数化预训练相比，SLTrain 几乎没有额外的参数和内存开销，但实现了与完全秩训练相当的性能。当与量化和逐层更新相结合时，SLTrain 可以将内存要求降低高达 73%。