Low-rank adaptation (LoRA) has become the default approach to fine-tune large
language models (LLMs) due to its significant reduction in trainable
parameters. However, trainable parameter demand for LoRA increases with
increasing model embedding dimensions, leading to high compute costs.
Additionally, its backward updates require storing high-dimensional
intermediate activations and optimizer states, demanding high peak GPU memory.
In this paper, we introduce large model fine-tuning via spectrally decomposed
low-dimensional adaptation (LaMDA), a novel approach to fine-tuning large
language models, which leverages low-dimensional adaptation to achieve
significant reductions in trainable parameters and peak GPU memory footprint.
LaMDA freezes a first projection matrix (PMA) in the adaptation path while
introducing a low-dimensional trainable square matrix, resulting in substantial
reductions in trainable parameters and peak GPU memory usage. LaMDA gradually
freezes a second projection matrix (PMB) during the early fine-tuning stages,
reducing the compute cost associated with weight updates to enhance parameter
efficiency further. We also present an enhancement, LaMDA++, incorporating a
``lite-weight" adaptive rank allocation for the LoRA path via normalized
spectrum analysis of pre-trained model weights. We evaluate LaMDA/LaMDA++
across various tasks, including natural language understanding with the GLUE
benchmark, text summarization, natural language generation, and complex
reasoning on different LLMs. Results show that LaMDA matches or surpasses the
performance of existing alternatives while requiring up to 17.7x fewer
parameter updates and up to 1.32x lower peak GPU memory usage during
fine-tuning. Code will be publicly available.

通过光谱分解低维适应性的大型模型微调，LaMDA 可以在减少可训练参数和峰值 GPU 内存使用的同时，达到与现有替代方案相当甚至更好的性能。