Image inpainting task refers to erasing unwanted pixels from images and
filling them in a semantically consistent and realistic way. Traditionally, the
pixels that are wished to be erased are defined with binary masks. From the
application point of view, a user needs to generate the masks for the objects
they would like to remove which can be time-consuming and prone to errors. In
this work, we are interested in an image inpainting algorithm that estimates
which object to be removed based on natural language input and also removes it,
simultaneously. For this purpose, first, we construct a dataset named
GQA-Inpaint for this task which will be released soon. Second, we present a
novel inpainting framework, Inst-Inpaint, that can remove objects from images
based on the instructions given as text prompts. We set various GAN and
diffusion-based baselines and run experiments on synthetic and real image
datasets. We compare methods with different evaluation metrics that measure the
quality and accuracy of the models and show significant quantitative and
qualitative improvements.

本论文提出了一种基于自然语言输入的图像修复算法，名为 Inst-Inpaint，可同时估计要移除的对象，并对其进行修复。为了支持该任务，我们首先构建了一个名为 GQA-Inpaint 的数据集。我们在合成和真实图像数据集上运行了各种 GAN 和扩散基线，使用不同的评估指标比较方法，并展示了显著的定量和定性改进。