Adapter-based parameter-efficient transfer learning has achieved exciting
results in vision-language models. Traditional adapter methods often require
training or fine-tuning, facing challenges such as insufficient samples or
resource limitations. While some methods overcome the need for training by
leveraging image modality cache and retrieval, they overlook the text
modality's importance and cross-modal cues for the efficient adaptation of
parameters in visual-language models. This work introduces a cross-modal
parameter-efficient approach named XMAdapter. XMAdapter establishes cache
models for both text and image modalities. It then leverages retrieval through
visual-language bimodal information to gather clues for inference. By
dynamically adjusting the affinity ratio, it achieves cross-modal fusion,
decoupling different modal similarities to assess their respective
contributions. Additionally, it explores hard samples based on differences in
cross-modal affinity and enhances model performance through adaptive adjustment
of sample learning intensity. Extensive experimental results on benchmark
datasets demonstrate that XMAdapter outperforms previous adapter-based methods
significantly regarding accuracy, generalization, and efficiency.

XMAdapter 是一种跨模态参数高效的适配器方法，通过建立文本和图像的缓存模型，并利用视觉 - 语言双模态信息进行检索以获得推理线索。通过动态调整关联比例实现跨模态融合，解耦不同模态相似性以评估其各自的贡献，并通过适应性调整样本学习强度来增强模型性能。实验结果表明，XMAdapter 在准确性、泛化能力和效率方面明显优于以前的基于适配器的方法。