As the adoption of large language models increases and the need for per-user
or per-task model customization grows, the parameter-efficient fine-tuning
(PEFT) methods, such as low-rank adaptation (LoRA) and its variants, incur
substantial storage and transmission costs. To further reduce stored
parameters, we introduce a "divide-and-share" paradigm that breaks the barriers
of low-rank decomposition across matrix dimensions, modules and layers by
sharing parameters globally via a \textit{vector bank}. As an instantiation of
the paradigm to LoRA, our proposed VB-LoRA composites \textit{all} the low-rank
matrices of LoRA from a shared \textit{vector bank} with a differentiable
top-$k$ admixture module. VB-LoRA achieves extreme parameter efficiency while
maintaining comparable or better performance compared to state-of-the-art PEFT
methods. Extensive experiments demonstrate the effectiveness of VB-LoRA on
natural language understanding, natural language generation, and instruction
tuning tasks. When fine-tuning the Llama2-13B model, VB-LoRA only uses 0.4\% of
LoRA's stored parameters yet attaining superior results. Our source code is
available at https://github.com/leo-yangli/VB-LoRA.

通过使用 “divide-and-share” 范式并引入向量库，VB-LoRA 实现了极高的参数效率同时保持与当前最先进 PEFT 方法相当甚至更好的性能。