As the size of Large Multi-Modal Models (LMMs) increases consistently, the
adaptation of these pre-trained models to specialized tasks has become a
computationally and memory-intensive challenge. Traditional fine-tuning methods
require isolated, exhaustive retuning for each new task, limiting the models'
versatility. Moreover, current efficient adaptation techniques often overlook
modality alignment, focusing only on the knowledge extraction of new tasks. To
tackle these issues, we introduce Multiway-Adapter, an innovative framework
incorporating an 'Alignment Enhancer' to deepen modality alignment, enabling
high transferability without tuning pre-trained parameters. Our method adds
fewer than 1.25\% of additional parameters to LMMs, exemplified by the BEiT-3
model in our study. This leads to superior zero-shot image-text retrieval
performance compared to fully fine-tuned models, while achieving up to a 57\%
reduction in fine-tuning time. Our approach offers a resource-efficient and
effective adaptation pathway for LMMs, broadening their applicability. The
source code is publicly available at:
https://github.com/longkukuhi/MultiWay-Adapter.

通过引入 Multiway-Adapter 框架和 'Alignment Enhancer' 来深化多模态对齐，我们提出了一种高效的适应路径，使得大型多模态模型具备高度的可迁移性，同时实现了 57% 的微调时间缩减，并在零样本图像 - 文本检索任务中表现出优异的性能。