With the advent of vision-language models (VLMs) that can perform in-context
and prompt-based learning, how can we design prompting approaches that robustly
generalize to distribution shift and can be used on novel classes outside the
support set of the prompts? In this work, we first define two types of
robustness to distribution shift on VLMs, namely, robustness on base classes
(the classes included in the support set of prompts) and robustness on novel
classes. Then, we study the robustness of existing in-context learning and
prompt learning approaches, where we find that prompt learning performs
robustly on test images from base classes, while it does not generalize well on
images from novel classes. We propose robust prompt learning by integrating
multiple-scale image features into the prompt, which improves both types of
robustness. Comprehensive experiments are conducted to study the defined
robustness on six benchmarks and show the effectiveness of our proposal.

本研究通过将多尺度图像特征集成到提示中，提出了一种对于分布偏移具有鲁棒性的提示学习方法，实验结果表明，这种方法在多个基准测试数据集上的鲁棒性和性能有所提高。