Inpainting involves filling in missing pixels or areas in an image, a crucial
technique employed in Mixed Reality environments for various applications,
particularly in Diminished Reality (DR) where content is removed from a user's
visual environment. Existing methods rely on digital replacement techniques
which necessitate multiple cameras and incur high costs. AR devices and
smartphones use ToF depth sensors to capture scene depth maps aligned with RGB
images. Despite speed and affordability, ToF cameras create imperfect depth
maps with missing pixels. To address the above challenges, we propose
Hierarchical Inpainting GAN (HI-GAN), a novel approach comprising three GANs in
a hierarchical fashion for RGBD inpainting. EdgeGAN and LabelGAN inpaint masked
edge and segmentation label images respectively, while CombinedRGBD-GAN
combines their latent representation outputs and performs RGB and Depth
inpainting. Edge images and particularly segmentation label images as auxiliary
inputs significantly enhance inpainting performance by complementary context
and hierarchical optimization. We believe we make the first attempt to
incorporate label images into inpainting process.Unlike previous approaches
requiring multiple sequential models and separate outputs, our work operates in
an end-to-end manner, training all three models simultaneously and
hierarchically. Specifically, EdgeGAN and LabelGAN are first optimized
separately and further optimized inside CombinedRGBD-GAN to enhance inpainting
quality. Experiments demonstrate that HI-GAN works seamlessly and achieves
overall superior performance compared with existing approaches.

我们提出了一种新的分层修复 GAN（HI-GAN），它由三个 GAN 组成，EdgeGAN 和 LabelGAN 分别修复边缘图像和分割标签图像，而 CombinedRGBD-GAN 则结合它们的潜在输出并进行 RGB 和深度修复，同时边缘图像和分割标签图像作为辅助输入显著增强了修复性能，并通过分层优化实现了端到端的操作，实验证明 HI-GAN 在整体上表现优于现有方法。