Visual prompting infuses visual information into the input image to adapt
models toward specific predictions and tasks. Recently, manually crafted
markers such as red circles are shown to guide the model to attend to a target
region on the image. However, these markers only work on models trained with
data containing those markers. Moreover, finding these prompts requires
guesswork or prior knowledge of the domain on which the model is trained. This
work circumvents manual design constraints by proposing to learn the visual
prompts for guiding the attention of vision transformers. The learned visual
prompt, added to any input image would redirect the attention of the
pre-trained vision transformer to its spatial location on the image.
Specifically, the prompt is learned in a self-supervised manner without
requiring annotations and without fine-tuning the vision transformer. Our
experiments demonstrate the effectiveness of the proposed optimization-based
visual prompting strategy across various pre-trained vision encoders.

通过在输入图像中引入视觉提示信息，本研究旨在为视觉变换器模型设计学习视觉提示，以引导其注意力集中在图像的特定区域，通过自监督学习的方式进行优化，实验结果表明该优化策略在各种预训练视觉编码器中的效果显著。