Text-guided image generation enables the creation of visual content from
textual descriptions. However, certain visual concepts cannot be effectively
conveyed through language alone. This has sparked a renewed interest in
utilizing the CLIP image embedding space for more visually-oriented tasks
through methods such as IP-Adapter. Interestingly, the CLIP image embedding
space has been shown to be semantically meaningful, where linear operations
within this space yield semantically meaningful results. Yet, the specific
meaning of these operations can vary unpredictably across different images. To
harness this potential, we introduce pOps, a framework that trains specific
semantic operators directly on CLIP image embeddings. Each pOps operator is
built upon a pretrained Diffusion Prior model. While the Diffusion Prior model
was originally trained to map between text embeddings and image embeddings, we
demonstrate that it can be tuned to accommodate new input conditions, resulting
in a diffusion operator. Working directly over image embeddings not only
improves our ability to learn semantic operations but also allows us to
directly use a textual CLIP loss as an additional supervision when needed. We
show that pOps can be used to learn a variety of photo-inspired operators with
distinct semantic meanings, highlighting the semantic diversity and potential
of our proposed approach.

借助文本引导的图像生成，通过 IP-Adapter 等方法利用 CLIP 图像嵌入空间从事面向视觉的任务，但线性操作在此空间内的具体含义在不同图像之间可能变化不可预测，为此，我们引入了 pOps 框架，在 CLIP 图像嵌入上直接训练特定的语义操作符，每个 pOps 操作符都基于预训练的扩散先验模型构建，pOps 不仅提高了我们学习语义操作的能力，还允许我们在需要时直接使用文本 CLIP 损失作为额外的监督，我们展示了 pOps 可以用于学习多种具有不同语义含义的受照片启发的操作符，突显了我们提出方法的语义多样性和潜力。