Multimodal large language models (MLLMs) fine-tuned with multimodal
instruction datasets have demonstrated remarkable capabilities in multimodal
tasks. However, fine-tuning all parameters of MLLMs has become challenging as
they usually contain billions of parameters. To address this issue, we study
parameter-efficient fine-tuning (PEFT) methods for MLLMs. We aim to identify
effective methods for enhancing the performance of MLLMs in scenarios where
only a limited number of parameters are trained. This paper conducts empirical
studies using four popular PEFT methods to fine-tune the LLM component of
open-source MLLMs. We present a comprehensive analysis that encompasses various
aspects, including the impact of PEFT methods on various models, parameters and
location of the PEFT module, size of fine-tuning data, model stability based on
PEFT methods, MLLM's generalization, and hallucination. We evaluated four PEFT
methods on seven datasets from two different categories: unseen and seen
datasets. Across all experiments, we show that the adapter is the
best-performing PEFT method. At the same time, fine-tuning the connector layers
leads to improved performance in most MLLMs. Code and data are available at
this https URL

使用四种常见的 PEFT 方法对开源 MLLMs 的 LLM 组件进行微调，比较它们在参数效率方面的表现，结果显示 adapter 是效果最好的 PEFT 方法。