The rapid advancements in Large Language Models (LLMs) have revolutionized
various natural language processing tasks. However, the substantial size of
LLMs presents significant challenges in training or fine-tuning. While
parameter-efficient approaches such as low-rank adaptation (LoRA) have gained
popularity, they often compromise performance compared to full-rank
fine-tuning. In this paper, we propose Outlier-weighed Layerwise Sampled
Low-Rank Projection (OwLore), a new memory-efficient fine-tuning approach,
inspired by the layerwise outlier distribution of LLMs, which dynamically
samples pre-trained layers to fine-tune instead of adding additional adaptors.
We first interpret the outlier phenomenon through the lens of Heavy-Tailed
Self-Regularization theory (HT-SR), discovering that layers with more outliers
tend to be more heavy-tailed and consequently better trained. Inspired by this
finding, OwLore strategically assigns higher sampling probabilities to layers
with more outliers to better leverage the knowledge stored in pre-trained LLMs.
To further mitigate the memory demands of fine-tuning, we integrate gradient
low-rank projection into our approach, which facilitates each layer to be
efficiently trained in a low-rank manner. By incorporating the efficient
characteristics of low-rank and optimal layerwise sampling, OwLore
significantly improves the memory-performance trade-off in LLM pruning. Our
extensive experiments across various architectures, including LLaMa2, LLaMa3,
and Mistral, demonstrate that OwLore consistently outperforms baseline
approaches, including full fine-tuning. Specifically, it achieves up to a 1.1%
average accuracy gain on the Commonsense Reasoning benchmark, a 3.0%
improvement on MMLU, and a notable 10% boost on MT-Bench, while being more
memory efficient. OwLore allows us to fine-tune LLaMa2-7B with only 21GB of
memory.

通过优化训练和内存利用，OwLore 方法在大型语言模型中实现了高效的微调和低内存占用。