We introduce a scalable framework for novel view synthesis from RGB-D images
with largely incomplete scene coverage. While generative neural approaches have
demonstrated spectacular results on 2D images, they have not yet achieved
similar photorealistic results in combination with scene completion where a
spatial 3D scene understanding is essential. To this end, we propose a
generative pipeline performing on a sparse grid-based neural scene
representation to complete unobserved scene parts via a learned distribution of
scenes in a 2.5D-3D-2.5D manner. We process encoded image features in 3D space
with a geometry completion network and a subsequent texture inpainting network
to extrapolate the missing area. Photorealistic image sequences can be finally
obtained via consistency-relevant differentiable rendering. Comprehensive
experiments show that the graphical outputs of our method outperform the state
of the art, especially within unobserved scene parts.

本研究提出了一种可扩展的框架，用于从具有大部分不完整场景覆盖的 RGB-D 图像中合成新视图。 通过一种稀疏网格神经场景表示学习得到的场景分布对未观察到的场景部分进行完成。 最终，本方法的图形输出在未观察到的场景部分方面优于现有技术。