We present a practical and robust deep learning solution for capturing and rendering novel views of complex real world scenes for virtual exploration. Previous approaches either require intractably dense view sampling or provide little to no guidance for how users should sample views of a scene to reliably render high-quality novel views. Instead, we propose an algorithm for view synthesis from an irregular grid of sampled views that first expands each sampled view into a local light field via a multiplane image (MPI) scene representation, then renders novel views by blending adjacent local light fields. We extend traditional plenoptic sampling theory to derive a bound that specifies precisely how densely users should sample views of a given scene when using our algorithm. In practice, we apply this bound to capture and render views of real world scenes that achieve the perceptual quality of Nyquist rate view sampling while using up to 4000x fewer views. We demonstrate our approach's practicality with an augmented reality smartphone app that guides users to capture input images of a scene and viewers that enable realtime virtual exploration on desktop and mobile platforms.

在虚拟探索中，我们提出了一种实用且强大的深度学习算法来捕捉和呈现复杂实景的新视图，该算法通过从不规则采样的视图网格中渲染新视图，并通过邻近局部光场进行混合来实现。我们将经典的光场采样理论扩展到适用于我们的算法的场景采样。实践中，我们利用该算法捕捉和渲染实景的视图，其感知质量达到Nyquist率视图采样的水平，并且使用的视图数量只有Nyquist率的1/4000。此外，我们还开发了一个增强现实智能手机应用程序，用于指导用户捕捉场景的输入图像，并在桌面和移动平台上实现了实时虚拟探索。

局部光场融合: 可行的视图综合及规定采样指南