We introduce a novel diffusion transformer, LazyDiffusion, that generates
partial image updates efficiently. Our approach targets interactive image
editing applications in which, starting from a blank canvas or an image, a user
specifies a sequence of localized image modifications using binary masks and
text prompts. Our generator operates in two phases. First, a context encoder
processes the current canvas and user mask to produce a compact global context
tailored to the region to generate. Second, conditioned on this context, a
diffusion-based transformer decoder synthesizes the masked pixels in a "lazy"
fashion, i.e., it only generates the masked region. This contrasts with
previous works that either regenerate the full canvas, wasting time and
computation, or confine processing to a tight rectangular crop around the mask,
ignoring the global image context altogether. Our decoder's runtime scales with
the mask size, which is typically small, while our encoder introduces
negligible overhead. We demonstrate that our approach is competitive with
state-of-the-art inpainting methods in terms of quality and fidelity while
providing a 10x speedup for typical user interactions, where the editing mask
represents 10% of the image.

我们介绍了一种新颖的扩散变换器 LazyDiffusion，可高效地生成部分图像更新。我们的方法针对交互式图像编辑应用，其中用户使用二进制掩码和文本提示指定一系列局部图像修改。我们的生成器分为两个阶段：首先，上下文编码器对当前画布和用户掩码进行处理，以生成与待生成区域相适应的简洁全局上下文。其次，在此上下文的条件下，基于扩散的变换器解码器以 “懒惰” 的方式合成掩码像素，即仅生成掩码区域。我们的解码器的运行时间与掩码大小成比例，而我们的编码器几乎不会引入额外开销。我们证明，我们的方法在质量和保真度方面与最先进的修复方法相当，同时为典型的用户交互提供 10 倍的加速，其中编辑掩码占图像的 10%。

懒惰扩散转换器用于交互式图像编辑

Lazy Diffusion Transformer for Interactive Image Editing

Recently, text-guided image manipulation has received increasing attention in
the research field of multimedia processing and computer vision due to its high
flexibility and controllability. Its goal is to semantically manipulate parts
of an input reference image according to the text descriptions. However, most
of the existing works have the following problems: (1) text-irrelevant content
cannot always be maintained but randomly changed, (2) the performance of image
manipulation still needs to be further improved, (3) only can manipulate
descriptive attributes. To solve these problems, we propose a novel image
manipulation method that interactively edits an image using complex text
instructions. It allows users to not only improve the accuracy of image
manipulation but also achieve complex tasks such as enlarging, dwindling, or
removing objects and replacing the background with the input image. To make
these tasks possible, we apply three strategies. First, the given image is
divided into text-relevant content and text-irrelevant content. Only the
text-relevant content is manipulated and the text-irrelevant content can be
maintained. Second, a super-resolution method is used to enlarge the
manipulation region to further improve the operability and to help manipulate
the object itself. Third, a user interface is introduced for editing the
segmentation map interactively to re-modify the generated image according to
the user's desires. Extensive experiments on the Caltech-UCSD Birds-200-2011
(CUB) dataset and Microsoft Common Objects in Context (MS COCO) datasets
demonstrate our proposed method can enable interactive, flexible, and accurate
image manipulation in real-time. Through qualitative and quantitative
evaluations, we show that the proposed model outperforms other state-of-the-art
methods.

本研究提出了一种基于复杂文本描述的图像编辑方法，在文本相关部分对输入的图像进行修改，并引入用户界面对生成的图像进行交互式重新修改，实现实时交互、灵活和准确的图像操作。

复杂文本指令下的交互式图像操作

Interactive Image Manipulation with Complex Text Instructions

Most existing text-to-image synthesis tasks are static single-turn
generation, based on pre-defined textual descriptions of images. To explore
more practical and interactive real-life applications, we introduce a new task
- Interactive Image Editing, where users can guide an agent to edit images via
multi-turn textual commands on-the-fly. In each session, the agent takes a
natural language description from the user as the input and modifies the image
generated in the previous turn to a new design, following the user description.
The main challenges in this sequential and interactive image generation task
are two-fold: 1) contextual consistency between a generated image and the
provided textual description; 2) step-by-step region-level modification to
maintain visual consistency across the generated image sequence in each
session. To address these challenges, we propose a novel Sequential Attention
Generative Adversarial Net-work (SeqAttnGAN), which applies a neural state
tracker to encode the previous image and the textual description in each turn
of the sequence, and uses a GAN framework to generate a modified version of the
image that is consistent with the preceding images and coherent with the
description. To achieve better region-specific refinement, we also introduce a
sequential attention mechanism into the model. To benchmark on the new task, we
introduce two new datasets, Zap-Seq and DeepFashion-Seq, which contain
multi-turn sessions with image-description sequences in the fashion domain.
Experiments on both datasets show that the proposed SeqAttnGANmodel outperforms
state-of-the-art approaches on the interactive image editing task across all
evaluation metrics including visual quality, image sequence coherence, and
text-image consistency.

提出了一种基于 SeqAttnGAN 模型的交互式图片编辑方法，该模型应用了神经状态跟踪器进行图像和文本描述的编码，并使用 GAN 框架生成与之前图像一致且符合描述的新版本图像，同时为了实现更好的区域特定细化引入了顺序注意机制。在新的任务基准上，实验表明所提出的 SeqAttnGAN 模型在所有评估度量上优于现有方法。