Vision-Language Models (VLMs) like CLIP have demonstrated remarkable
applicability across a variety of downstream tasks, including zero-shot image
classification. Recently, the use of prompts or adapters for efficient transfer
learning has gained significant attention for effectively adapting to
downstream tasks. However, the roles of vision and text prompts, as well as
adapters in terms of generalization and transfer difficulty, have been
overlooked, limiting performance on unseen tasks. In this paper, we empirically
analyze how VLMs behave when using vision and text prompts, adapters, and a
combination of these components, marking a novel exploration by our study. Our
observations find that utilizing vision prompts for class separability and text
adapters for task adaptation is crucial for adaptability and generalizability.
Moreover, to improve generalization across every domain, we propose an adaptive
ensemble method that effectively combines the general knowledge of VLMs with
task-specific knowledge according to transfer difficulty. Upon experimenting
with extensive benchmarks, our method consistently outperforms all baselines,
particularly on unseen tasks, demonstrating the effectiveness of our proposed
approach.

利用视觉提示进行类别可分性和使用文本适配器进行任务适应的结合对于适应性和泛化性至关重要；我们提出了一种自适应集成方法，根据迁移难度有效地结合了 VLMs 的通用知识和任务特定知识，该方法在广泛的基准测试中始终优于所有基准线，尤其是在未知任务上表现出了其有效性。