While developing a new vision-language LLM (VL-LLM) by pre-training on
tremendous image-text pairs from scratch can be exceedingly resource-consuming,
connecting an existing LLM with a comparatively lightweight visual prompt
generator (VPG) becomes a feasible paradigm. However, further tuning the VPG
part of the VL-LLM still suffers from indispensable computational costs, i.e.,
requiring thousands of GPU hours and millions of training data. One alternative
solution is to transfer an existing VPG from any existing VL-LLMs for the
target VL-LLM.
In this work, we for the first time investigate the VPG transferability
across LLMs, and explore a solution to reduce the cost of VPG transfer. We
first study the VPG transfer across different LLM sizes (e.g., small-to-large),
and across different LLM types, through which we diagnose the key factors to
maximize the transfer efficiency. Based on our observation, we design a
two-stage transfer framework named VPGTrans, which is simple yet highly
effective. Through extensive experiments, we demonstrate that VPGTrans helps
significantly speed up the transfer learning process without compromising
performance. Remarkably, it helps achieve the VPG transfer from BLIP-2
OPT$_\text{2.7B}$ to BLIP-2 OPT$_\text{6.7B}$ with over 10 times speed-up and
10.7% training data compared with connecting a VPG to OPT$_\text{6.7B}$ from
scratch. Further, a series of intriguing findings and potential rationales
behind them are provided and discussed. Finally, we showcase the practical
value of our VPGTrans approach, by customizing two novel VL-LLMs, including
VL-LLaMA and VL-Vicuna, with recently released LLaMA and Vicuna LLMs.

本文旨在探究现有 VPG 在 VL-LLMs 之间的可迁移性，并通过提出的 VPGTrans 方案，成功在不牺牲性能的情况下将 VPG 从 BLIP-2 OPT $_{2.7B}$ 迁移到了 BLIP-2 OPT $_{6.7B}$，在迁移过程中节省了 10 倍的时间和 10.7％ 的训练数据。