The choice of input text prompt plays a critical role in the performance of
Vision-Language Pretrained (VLP) models such as CLIP. We present APoLLo, a
unified multi-modal approach that combines Adapter and Prompt learning for
Vision-Language models. Our method is designed to substantially improve the
generalization capabilities of VLP models when they are fine-tuned in a
few-shot setting. We introduce trainable cross-attention-based adapter layers
in conjunction with vision and language encoders to strengthen the alignment
between the two modalities. We enforce consistency between the respective
encoder branches (receiving augmented inputs) to prevent overfitting in
downstream tasks. Our method is evaluated on three representative tasks:
generalization to novel classes, cross-dataset evaluation, and unseen domain
shifts. In practice, APoLLo achieves a relative gain up to 6.03% over MaPLe
(SOTA) on novel classes for 10 diverse image recognition datasets.

APoLLo 是一种统一的多模态方法，结合了 Adapter 和 Prompt 学习用于增强 Vision-Language 模型在 few-shot 环境下的泛化能力，通过引入可训练的交叉注意力适配器层和视觉与语言编码器来加强两种模态之间的对齐，并通过确保编码器分支之间的一致性来防止过拟合，在新颖类别的 10 个不同图像识别数据集上，APoLLo 相对于 MaLPe (SOTA) 实现了高达 6.03% 的相对增益。