In this work, we study the task of sketch-guided image inpainting. Unlike the
well-explored natural language-guided image inpainting, which excels in
capturing semantic details, the relatively less-studied sketch-guided
inpainting offers greater user control in specifying the object's shape and
pose to be inpainted. As one of the early solutions to this task, we introduce
a novel partial discrete diffusion process (PDDP). The forward pass of the PDDP
corrupts the masked regions of the image and the backward pass reconstructs
these masked regions conditioned on hand-drawn sketches using our proposed
sketch-guided bi-directional transformer. The proposed novel transformer module
accepts two inputs -- the image containing the masked region to be inpainted
and the query sketch to model the reverse diffusion process. This strategy
effectively addresses the domain gap between sketches and natural images,
thereby, enhancing the quality of inpainting results. In the absence of a
large-scale dataset specific to this task, we synthesize a dataset from the
MS-COCO to train and extensively evaluate our proposed framework against
various competent approaches in the literature. The qualitative and
quantitative results and user studies establish that the proposed method
inpaints realistic objects that fit the context in terms of the visual
appearance of the provided sketch. To aid further research, we have made our
code publicly available at this https URL .

本研究工作中，我们研究了基于草图引导的图像修复任务。与在捕捉语义细节方面表现出色的经过深入研究的自然语言引导图像修复不同，相对较少研究的草图引导修复提供了更强的用户控制力，可指定待修复对象的形状和姿势。作为这个任务的早期解决方案之一，我们引入了一种新颖的部分离散扩散过程（PDDP）。PDDP 的正向传递在图像的掩蔽区域损坏，而反向传递使用我们提出的草图引导双向转换器以依赖于手绘草图来重建这些掩蔽区域。所提出的新颖转换器模块接受两个输入 —— 包含待修复区域的图像和用于建模逆扩散过程的查询草图。这一策略有效地解决了草图与自然图像之间的领域差异，从而提高了修复结果的质量。在缺乏针对此任务的大规模数据集的情况下，我们从 MS-COCO 合成了一个数据集，用于对我们提出的框架进行训练和广泛评估，与文献中的各种竞争方法进行对比。定性和定量结果以及用户研究表明，所提出的方法可以修复与提供的草图的视觉外观相符的逼真对象。为了促进进一步的研究，我们已将我们的代码公开提供在此 https URL 。