We propose a generative model that, given a coarsely edited image,
synthesizes a photorealistic output that follows the prescribed layout. Our
method transfers fine details from the original image and preserves the
identity of its parts. Yet, it adapts it to the lighting and context defined by
the new layout. Our key insight is that videos are a powerful source of
supervision for this task: objects and camera motions provide many observations
of how the world changes with viewpoint, lighting, and physical interactions.
We construct an image dataset in which each sample is a pair of source and
target frames extracted from the same video at randomly chosen time intervals.
We warp the source frame toward the target using two motion models that mimic
the expected test-time user edits. We supervise our model to translate the
warped image into the ground truth, starting from a pretrained diffusion model.
Our model design explicitly enables fine detail transfer from the source frame
to the generated image, while closely following the user-specified layout. We
show that by using simple segmentations and coarse 2D manipulations, we can
synthesize a photorealistic edit faithful to the user's input while addressing
second-order effects like harmonizing the lighting and physical interactions
between edited objects.

我们提出了一个生成模型，根据粗略编辑的图片合成一个遵循预定布局的逼真输出，并从原始图像中转移细节，同时保留其部分的身份信息，并将其调整到由新布局定义的光照和上下文。