This article focuses on large language models (LLMs) fine-tuning in the scarce data regime (also known as the "few-shot" learning setting). We propose a method to increase the generalization capabilities of LLMs based on neural network subspaces. This optimization method, recently introduced in computer vision, aims to improve model generalization by identifying wider local optima through the joint optimization of an entire simplex of models in parameter space. Its adaptation to massive, pretrained transformers, however, poses some challenges. First, their considerable number of parameters makes it difficult to train several models jointly, and second, their deterministic parameter initialization schemes make them unfit for the subspace method as originally proposed. We show in this paper that "Parameter Efficient Fine-Tuning" (PEFT) methods, however, are perfectly compatible with this original approach, and propose to learn entire simplex of continuous prefixes. We test our method on a variant of the GLUE benchmark adapted to the few-shot learning setting, and show that both our contributions jointly lead to a gain in average performances compared to sota methods. The implementation can be found at the following link: https://github.com/Liloulou/prefix_subspace

在稀缺数据情况下，本研究聚焦于大规模语言模型（LLMs）的精细调整，提出了一种基于神经网络子空间的方法，通过在参数空间中联合优化一个模型单纯形来增加LLMs的泛化能力。使用“Parameter Efficient Fine-Tuning”（PEFT）方法来学习连续前缀的整个单纯形，实验证明这种方法与原始方法相兼容，同时在适应few-shot学习设置的GLUE基准的变种上取得了优于sota方法的平均性能提升。

通过学习前缀子空间提高大型语言模型的泛化能力