We introduce a language-grounded visual prompting method to adapt the visual
encoder of vision-language models for downstream tasks. By capitalizing on
language integration, we devise a parameter-efficient strategy to adjust the
input of the visual encoder, eliminating the need to modify or add to the
model's parameters. Due to this design choice, our algorithm can operate even
in black-box scenarios, showcasing adaptability in situations where access to
the model's parameters is constrained. We will empirically demonstrate that,
compared to prior art, grounding visual prompts with language enhances both the
accuracy and speed of adaptation. Moreover, our algorithm excels in
base-to-novel class generalization, overcoming limitations of visual prompting
and exhibiting the capacity to generalize beyond seen classes. We thoroughly
assess and evaluate our method across a variety of image recognition datasets,
such as EuroSAT, UCF101, DTD, and CLEVR, spanning different learning
situations, including few-shot learning, base-to-novel class generalization,
and transfer learning.

通过语言引导的视觉提示方法，我们使视觉语言模型的视觉编码器适应下游任务，从而提高适应性和泛化性能。

LaViP: 语言驱动的视觉提示

LaViP:Language-Grounded Visual Prompts

In recent years, soft prompt learning methods have been proposed to fine-tune
large-scale vision-language pre-trained models for various downstream tasks.
These methods typically combine learnable textual tokens with class tokens as
input for models with frozen parameters. However, they often employ a single
prompt to describe class contexts, failing to capture categories' diverse
attributes adequately. This study introduces the Partitioned Multi-modal Prompt
(PMPO), a multi-modal prompting technique that extends the soft prompt from a
single learnable prompt to multiple prompts. Our method divides the visual
encoder depths and connects learnable prompts to the separated visual depths,
enabling different prompts to capture the hierarchical contextual depths of
visual representations. Furthermore, to maximize the advantages of multi-prompt
learning, we incorporate prior information from manually designed templates and
learnable multi-prompts, thus improving the generalization capabilities of our
approach. We evaluate the effectiveness of our approach on three challenging
tasks: new class generalization, cross-dataset evaluation, and domain
generalization. For instance, our method achieves a $79.28$ harmonic mean,
averaged over 11 diverse image recognition datasets ($+7.62$ compared to CoOp),
demonstrating significant competitiveness compared to state-of-the-art
prompting methods.

本研究提出了分区多模态提示方法（PMPO），将软提示从单个可学习提示扩展到多个提示，以捕获视觉表示的分层上下文深度。此外，我们结合手动设计的模板和可学习的多提示，以提高方法的泛化能力，并在新类别概括、跨数据集评估和领域泛化等三项具有挑战性的任务上验证了它的有效性。