Large Language Models (LLMs) have recently gained popularity due to their
impressive few-shot performance across various downstream tasks. However,
fine-tuning all parameters and storing a unique model for each downstream task
or domain becomes impractical because of the massive size of checkpoints (e.g.,
350GB in GPT-3). Current literature, such as LoRA, showcases the potential of
low-rank modifications to the original weights of an LLM, enabling efficient
adaptation and storage for task-specific models. These methods can reduce the
number of parameters needed to fine-tune an LLM by several orders of magnitude.
Yet, these methods face two primary limitations: 1) the parameter reduction is
lower-bounded by the rank one decomposition, and 2) the extent of reduction is
heavily influenced by both the model architecture and the chosen rank. For
instance, in larger models, even a rank one decomposition might exceed the
number of parameters truly needed for adaptation. In this paper, we introduce
NOLA, which overcomes the rank one lower bound present in LoRA. It achieves
this by re-parameterizing the low-rank matrices in LoRA using linear
combinations of randomly generated matrices (basis) and optimizing the linear
mixture coefficients only. This approach allows us to decouple the number of
trainable parameters from both the choice of rank and the network architecture.
We present adaptation results using GPT-2 and ViT in natural language and
computer vision tasks. NOLA performs as well as, or better than models with
equivalent parameter counts. Furthermore, we demonstrate that we can halve the
parameters in larger models compared to LoRA with rank one, without sacrificing
performance.

通过重新参数化使用线性组合随机生成的矩阵（基）的低秩矩阵，并仅优化线性混合系数，NOLA 将创新地克服了 LoRA 中的秩一下界，与等效参数计数的模型相比，NOLA 在自然语言处理和计算机视觉任务中表现得很好，并且证明能在更大的模型中减少一半的参数而不降低性能。