Recently, we have observed that Large Multi-modal Models (LMMs) are revolutionizing the way machines interact with the world, unlocking new possibilities across various multi-modal applications. To adapt LMMs for downstream tasks, parameter-efficient fine-tuning (PEFT) which only trains additional prefix tokens or modules, has gained popularity. Nevertheless, there has been little analysis of how PEFT works in LMMs. In this paper, we delve into the strengths and weaknesses of each tuning strategy, shifting the focus from the efficiency typically associated with these approaches. We first discover that model parameter tuning methods such as LoRA and Adapters distort the feature representation space learned during pre-training and limit the full utilization of pre-trained knowledge. We also demonstrate that prefix-tuning excels at preserving the representation space, despite its lower performance on downstream tasks. These findings suggest a simple two-step PEFT strategy called Prefix-Tuned PEFT (PT-PEFT), which successively performs prefix-tuning and then PEFT (i.e., Adapter, LoRA), combines the benefits of both. Experimental results show that PT-PEFT not only improves performance in image captioning and visual question answering compared to vanilla PEFT methods but also helps preserve the representation space of the four pre-trained models.

本文探讨了大型多模态模型（LMMs）的参数高效微调（PEFT）方法，尤其是前缀调优如何在保留预训练表示空间方面表现优异。研究表明，传统的调优方法如LoRA和Adapters会扭曲预训练时学习的特征表示空间，而前缀调优能更好地保持这种表示，提出的前缀调优PEFT（PT-PEFT）策略显著提升了图像描述和视觉问答任务的性能。

保存预训练表示空间：大型多模态模型中前缀调优的有效性