Sketch-based image editing aims to synthesize and modify photos based on the
structural information provided by the human-drawn sketches. Since sketches are
difficult to collect, previous methods mainly use edge maps instead of sketches
to train models (referred to as edge-based models). However, sketches display
great structural discrepancy with edge maps, thus failing edge-based models.
Moreover, sketches often demonstrate huge variety among different users,
demanding even higher generalizability and robustness for the editing model to
work. In this paper, we propose Deep Plastic Surgery, a novel, robust and
controllable image editing framework that allows users to interactively edit
images using hand-drawn sketch inputs. We present a sketch refinement strategy,
as inspired by the coarse-to-fine drawing process of the artists, which we show
can help our model well adapt to casual and varied sketches without the need
for real sketch training data. Our model further provides a refinement level
control parameter that enables users to flexibly define how "reliable" the
input sketch should be considered for the final output, balancing between
sketch faithfulness and output verisimilitude (as the two goals might
contradict if the input sketch is drawn poorly). To achieve the multi-level
refinement, we introduce a style-based module for level conditioning, which
allows adaptive feature representations for different levels in a singe
network. Extensive experimental results demonstrate the superiority of our
approach in improving the visual quality and user controllablity of image
editing over the state-of-the-art methods.

本文提出了一种基于手绘草图输入的图像编辑框架 Deep Plastic Surgery，并介绍了可以使模型适应不同草图的润色策略、调整草图精度的控制参数、以及用于多级润色的基于样式的模块。实验结果表明，与现有方法相比，该方法能够提高图像编辑的视觉质量和用户可控性。