Image inpainting is an ill-posed problem to recover missing or damaged image
content based on incomplete images with masks. Previous works usually predict
the auxiliary structures (e.g., edges, segmentation and contours) to help fill
visually realistic patches in a multi-stage fashion. However, imprecise
auxiliary priors may yield biased inpainted results. Besides, it is
time-consuming for some methods to be implemented by multiple stages of complex
neural networks. To solve this issue, we develop an end-to-end multi-modality
guided transformer network, including one inpainting branch and two auxiliary
branches for semantic segmentation and edge textures. Within each transformer
block, the proposed multi-scale spatial-aware attention module can learn the
multi-modal structural features efficiently via auxiliary denormalization.
Different from previous methods relying on direct guidance from biased priors,
our method enriches semantically consistent context in an image based on
discriminative interplay information from multiple modalities. Comprehensive
experiments on several challenging image inpainting datasets show that our
method achieves state-of-the-art performance to deal with various
regular/irregular masks efficiently.

该论文介绍了一种用于图像修复的新型深度神经网络，该网络包含了一个修复分支和两个辅助分支，能够有效地利用多模式定位和语义信息，实现对各种正 / 不规则图像遮挡的修复，达到了最好的效果。