Pre-trained contrastive vision-language models have demonstrated remarkable
performance across a wide range of tasks. However, they often struggle on
fine-trained datasets with categories not adequately represented during
pre-training, which makes adaptation necessary. Recent works have shown
promising results by utilizing samples from web-scale databases for
retrieval-augmented adaptation, especially in low-data regimes. Despite the
empirical success, understanding how retrieval impacts the adaptation of
vision-language models remains an open research question. In this work, we
adopt a reflective perspective by presenting a systematic study to understand
the roles of key components in retrieval-augmented adaptation. We unveil new
insights on uni-modal and cross-modal retrieval and highlight the critical role
of logit ensemble for effective adaptation. We further present theoretical
underpinnings that directly support our empirical observations.

采用对比式视觉 - 语言模型，通过检索增强的适应性实现了卓越的性能，在了解检索对视觉 - 语言模型适应性的影响方面展示了新的见解，并强调了对有效适应至关重要的对数集成的关键作用。