The Stable Diffusion model is a prominent text-to-image generation model that
relies on a text prompt as its input, which is encoded using the Contrastive
Language-Image Pre-Training (CLIP). However, text prompts have limitations when
it comes to incorporating implicit information from reference images. Existing
methods have attempted to address this limitation by employing expensive
training procedures involving millions of training samples for image-to-image
generation. In contrast, this paper demonstrates that the CLIP model, as
utilized in Stable Diffusion, inherently possesses the ability to
instantaneously convert images into text prompts. Such an image-to-prompt
conversion can be achieved by utilizing a linear projection matrix that is
calculated in a closed form. Moreover, the paper showcases that this capability
can be further enhanced by either utilizing a small amount of similar-domain
training data (approximately 100 images) or incorporating several online
training steps (around 30 iterations) on the reference images. By leveraging
these approaches, the proposed method offers a simple and flexible solution to
bridge the gap between images and text prompts. This methodology can be applied
to various tasks such as image variation and image editing, facilitating more
effective and seamless interaction between images and textual prompts.

本文提出的方法，可以将图片转化为文本提示，从而解决了基于文本提示生成图片时难以整合隐含信息的问题。此方法可以简单灵活地处理各种任务，如图像变化和图像编辑。