Visual In-Context Learning (ICL) has emerged as a promising research area due
to its capability to accomplish various tasks with limited example pairs
through analogical reasoning. However, training-based visual ICL has
limitations in its ability to generalize to unseen tasks and requires the
collection of a diverse task dataset. On the other hand, existing methods in
the inference-based visual ICL category solely rely on textual prompts, which
fail to capture fine-grained contextual information from given examples and can
be time-consuming when converting from images to text prompts. To address these
challenges, we propose Analogist, a novel inference-based visual ICL approach
that exploits both visual and textual prompting techniques using a
text-to-image diffusion model pretrained for image inpainting. For visual
prompting, we propose a self-attention cloning (SAC) method to guide the
fine-grained structural-level analogy between image examples. For textual
prompting, we leverage GPT-4V's visual reasoning capability to efficiently
generate text prompts and introduce a cross-attention masking (CAM) operation
to enhance the accuracy of semantic-level analogy guided by text prompts. Our
method is out-of-the-box and does not require fine-tuning or optimization. It
is also generic and flexible, enabling a wide range of visual tasks to be
performed in an in-context manner. Extensive experiments demonstrate the
superiority of our method over existing approaches, both qualitatively and
quantitatively.

通过使用在图像修复中预训练的文本到图像扩散模型，我们提出了 Analogist 作为一种新颖的推理型视觉上下文学习方法，旨在利用视觉和文本提示技术，提高任务的泛化能力和上下文感知能力，从而在各种视觉任务中取得优越性能。