Recent advances in diffusion models enable many powerful instruments for
image editing. One of these instruments is text-driven image manipulations:
editing semantic attributes of an image according to the provided text
description. % Popular text-conditional diffusion models offer various
high-quality image manipulation methods for a broad range of text prompts.
Existing diffusion-based methods already achieve high-quality image
manipulations for a broad range of text prompts. However, in practice, these
methods require high computation costs even with a high-end GPU. This greatly
limits potential real-world applications of diffusion-based image editing,
especially when running on user devices.
In this paper, we address efficiency of the recent text-driven editing
methods based on unconditional diffusion models and develop a novel algorithm
that learns image manipulations 4.5-10 times faster and applies them 8 times
faster. We carefully evaluate the visual quality and expressiveness of our
approach on multiple datasets using human annotators. Our experiments
demonstrate that our algorithm achieves the quality of much more expensive
methods. Finally, we show that our approach can adapt the pretrained model to
the user-specified image and text description on the fly just for 4 seconds. In
this setting, we notice that more compact unconditional diffusion models can be
considered as a rational alternative to the popular text-conditional
counterparts.

该论文研究了基于扩散模型的无条件文本驱动图像编辑方法的效率，并开发了一种新算法，可以快速学习和应用图像操作，从而提高实现应用的潜力。

基于无条件扩散模型的实时文本驱动图像操作

Towards Real-time Text-driven Image Manipulation with Unconditional  Diffusion Models

Text-driven image manipulation remains challenging in training or inference
flexibility. Conditional generative models depend heavily on expensive
annotated training data. Meanwhile, recent frameworks, which leverage
pre-trained vision-language models, are limited by either per text-prompt
optimization or inference-time hyper-parameters tuning. In this work, we
propose a novel framework named \textit{DeltaEdit} to address these problems.
Our key idea is to investigate and identify a space, namely delta image and
text space that has well-aligned distribution between CLIP visual feature
differences of two images and CLIP textual embedding differences of source and
target texts. Based on the CLIP delta space, the DeltaEdit network is designed
to map the CLIP visual features differences to the editing directions of
StyleGAN at training phase. Then, in inference phase, DeltaEdit predicts the
StyleGAN's editing directions from the differences of the CLIP textual
features. In this way, DeltaEdit is trained in a text-free manner. Once
trained, it can well generalize to various text prompts for zero-shot inference
without bells and whistles. Code is available at
this https URL

本文提出了一个名为 DeltaEdit 的新框架，通过以 CLIP Delta 空间为基础来进行原始图像和目标图像之间的编辑，从而实现了基于文本的图像编辑的零样本推理。

DeltaEdit: 探索文本无关的训练用于以文本为驱动的图像处理

DeltaEdit: Exploring Text-free Training for Text-Driven Image  Manipulation

Recently, GAN inversion methods combined with Contrastive Language-Image
Pretraining (CLIP) enables zero-shot image manipulation guided by text prompts.
However, their applications to diverse real images are still difficult due to
the limited GAN inversion capability. Specifically, these approaches often have
difficulties in reconstructing images with novel poses, views, and highly
variable contents compared to the training data, altering object identity, or
producing unwanted image artifacts. To mitigate these problems and enable
faithful manipulation of real images, we propose a novel method, dubbed
DiffusionCLIP, that performs text-driven image manipulation using diffusion
models. Based on full inversion capability and high-quality image generation
power of recent diffusion models, our method performs zero-shot image
manipulation successfully even between unseen domains and takes another step
towards general application by manipulating images from a widely varying
ImageNet dataset. Furthermore, we propose a novel noise combination method that
allows straightforward multi-attribute manipulation. Extensive experiments and
human evaluation confirmed robust and superior manipulation performance of our
methods compared to the existing baselines. Code is available at
this https URL

该研究提出了一种名为 DiffusionCLIP 的新方法，该方法使用扩散模型进行文本驱动的图像操作，比现有基线表现更为优异，并允许简便的多属性操作。