Text-conditioned image editing has recently attracted considerable interest.
However, most methods are currently either limited to specific editing types
(e.g., object overlay, style transfer), or apply to synthetically generated
images, or require multiple input images of a common object. In this paper we
demonstrate, for the very first time, the ability to apply complex (e.g.,
non-rigid) text-guided semantic edits to a single real image. For example, we
can change the posture and composition of one or multiple objects inside an
image, while preserving its original characteristics. Our method can make a
standing dog sit down or jump, cause a bird to spread its wings, etc. -- each
within its single high-resolution natural image provided by the user. Contrary
to previous work, our proposed method requires only a single input image and a
target text (the desired edit). It operates on real images, and does not
require any additional inputs (such as image masks or additional views of the
object). Our method, which we call "Imagic", leverages a pre-trained
text-to-image diffusion model for this task. It produces a text embedding that
aligns with both the input image and the target text, while fine-tuning the
diffusion model to capture the image-specific appearance. We demonstrate the
quality and versatility of our method on numerous inputs from various domains,
showcasing a plethora of high quality complex semantic image edits, all within
a single unified framework.

本文提出了一种名为 “Imagic” 的方法，该方法仅需要单个输入图像和目标文本，就能针对单张高分辨率的真实图像进行复杂的文本引导语义编辑，其采用预训练的文本到图像扩散模型进行柔性编辑，不需任何额外的输入或附加视图，从而展示了各个领域的高质量复杂语义图像编辑.

基于扩散模型的文本实时图像编辑技术 Imagic

Imagic: Text-Based Real Image Editing with Diffusion Models

We present a method for zero-shot, text-driven appearance manipulation in
natural images and videos. Given an input image or video and a target text
prompt, our goal is to edit the appearance of existing objects (e.g., object's
texture) or augment the scene with visual effects (e.g., smoke, fire) in a
semantically meaningful manner. We train a generator using an internal dataset
of training examples, extracted from a single input (image or video and target
text prompt), while leveraging an external pre-trained CLIP model to establish
our losses. Rather than directly generating the edited output, our key idea is
to generate an edit layer (color+opacity) that is composited over the original
input. This allows us to constrain the generation process and maintain high
fidelity to the original input via novel text-driven losses that are applied
directly to the edit layer. Our method neither relies on a pre-trained
generator nor requires user-provided edit masks. We demonstrate localized,
semantic edits on high-resolution natural images and videos across a variety of
objects and scenes.

本文介绍了一种用于零样本、文本驱动外观操作的方法，利用内部数据集训练生成器，结合外部的 CLIP 模型进行损失计算，并通过生成编辑层的方式实现色彩与透明度的操作来实现高保真的自然图像和视频语义编辑。