Differentiable volumetric rendering is a powerful paradigm for 3D reconstruction and novel view synthesis. However, standard volume rendering approaches struggle with degenerate geometries in the case of limited viewpoint diversity, a common scenario in robotics applications. In this work, we propose to use the multi-view photometric objective from the self-supervised depth estimation literature as a geometric regularizer for volumetric rendering, significantly improving novel view synthesis without requiring additional information. Building upon this insight, we explore the explicit modeling of scene geometry using a generalist Transformer, jointly learning a radiance field as well as depth and light fields with a set of shared latent codes. We demonstrate that sharing geometric information across tasks is mutually beneficial, leading to improvements over single-task learning without an increase in network complexity. Our DeLiRa architecture achieves state-of-the-art results on the ScanNet benchmark, enabling high quality volumetric rendering as well as real-time novel view and depth synthesis in the limited viewpoint diversity setting.

本文提出使用自监督深度估计文献中的多视角光度目标作为几何正则化器来对体积渲染进行改进，并使用变压器来显式建模场景几何形状和深度，并展示了这种方法的性能。通过共享潜在编码信息，无需增加网络复杂度，越过单任务学习的局限性，实现了针对受限视角多样性的高质量体积渲染和实时新视角和深度合成的任务，并在ScanNet基准测试中实现了最先进的结果。

DeLiRa: 自监督深度、光照和辐射场