Recent works in image inpainting have shown that structural information plays
an important role in recovering visually pleasing results. In this paper, we
propose an end-to-end architecture composed of two parallel UNet-based streams:
a main stream (MS) and a structure stream (SS). With the assistance of SS, MS
can produce plausible results with reasonable structures and realistic details.
Specifically, MS reconstructs detailed images by inferring missing structures
and textures simultaneously, and SS restores only missing structures by
processing the hierarchical information from the encoder of MS. By interacting
with SS in the training process, MS can be implicitly encouraged to exploit
structural cues. In order to help SS focus on structures and prevent textures
in MS from being affected, a gated unit is proposed to depress
structure-irrelevant activations in the information flow between MS and SS.
Furthermore, the multi-scale structure feature maps in SS are utilized to
explicitly guide the structure-reasonable image reconstruction in the decoder
of MS through the fusion block. Extensive experiments on CelebA, Paris
StreetView and Places2 datasets demonstrate that our proposed method
outperforms state-of-the-art methods.

本文介绍了一种基于 UNet 的双流终端到端架构，用于图像修复。其中，结构流辅助主流产生了合理结构和真实细节的结果。实验结果表明，该方法优于当前最先进的方法。