This paper demonstrates that a progressively aligned language model can
effectively bridge frozen vision encoders and large language models (LLMs).
While the fundamental architecture and pre-training methods of vision encoders
and LLMs have been extensively studied, the architecture and training strategy
of vision-language adapters vary significantly across recent works. Our
research undertakes a thorough exploration of the state-of-the-art perceiver
resampler architecture and builds a strong baseline. However, we observe that
the vision-language alignment with perceiver resampler exhibits slow
convergence and limited scalability with a lack of direct supervision. To
address this issue, we propose PaLM2-VAdapter, employing a progressively
aligned language model as the vision-language adapter. Compared to the strong
baseline with perceiver resampler, our method empirically shows faster
convergence, higher performance, and stronger scalability. Extensive
experiments across various Visual Question Answering (VQA) and captioning tasks
on both images and videos demonstrate that our model exhibits state-of-the-art
visual understanding and multi-modal reasoning capabilities. Notably, our
method achieves these advancements with 30~70% fewer parameters than the
state-of-the-art large vision-language models, marking a significant efficiency
improvement.

我们的研究表明，渐进对齐语言模型能够有效地连接冻结的视觉编码器和大型语言模型。通过采用渐进对齐的语言模型作为视觉 - 语言适配器，我们的方法在收敛速度、性能和可扩展性方面都超过了现有的方法，并且使用的参数数量比现有的大规模视觉 - 语言模型减少了 30~70%，从而实现了显著的效率提升。