This paper introduces a novel method of Progressive Low Rank Decomposition
(PLRD) tailored for the compression of large language models. Our approach
leverages a pre-trained model, which is then incrementally decompressed to
smaller sizes using progressively lower ranks. This method allows for
significant reductions in computational overhead and energy consumption, as
subsequent models are derived from the original without the need for retraining
from scratch. We detail the implementation of PLRD, which strategically
decreases the tensor ranks, thus optimizing the trade-off between model
performance and resource usage. The efficacy of PLRD is demonstrated through
extensive experiments showing that models trained with PLRD method on only 1B
tokens maintain comparable performance with traditionally trained models while
using 0.1% of the tokens. The versatility of PLRD is highlighted by its ability
to generate multiple model sizes from a single foundational model, adapting
fluidly to varying computational and memory budgets. Our findings suggest that
PLRD could set a new standard for the efficient scaling of LLMs, making
advanced AI more feasible on diverse platforms.

本文介绍了一种新颖的渐进低秩分解（PLRD）方法，特别适用于大型语言模型的压缩。我们的方法利用预训练模型，然后通过逐步降低秩来将其解压缩为更小的尺寸。这种方法减少了计算开销和能源消耗，因为后续模型是基于原始模型而不需要重新训练。我们详细介绍了 PLRD 的实现方式，策略性地降低了张量秩，从而在模型性能和资源使用之间优化了权衡。通过大量实验展示了 PLRD 的有效性，表明使用 PLRD 方法仅使用 1B 标记训练的模型在性能上与传统训练的模型相当，同时只使用了 0.1％的标记。PLRD 的多功能性通过能够从单个基础模型生成多个模型尺寸，以适应不同的计算和内存预算。我们的研究结果表明，PLRD 可能为 LLM 的高效扩展设定了新的标准，从而在各种平台上使先进的人工智能更加可行。