We present MVD-Fusion: a method for single-view 3D inference via generative
modeling of multi-view-consistent RGB-D images. While recent methods pursuing
3D inference advocate learning novel-view generative models, these generations
are not 3D-consistent and require a distillation process to generate a 3D
output. We instead cast the task of 3D inference as directly generating
mutually-consistent multiple views and build on the insight that additionally
inferring depth can provide a mechanism for enforcing this consistency.
Specifically, we train a denoising diffusion model to generate multi-view RGB-D
images given a single RGB input image and leverage the (intermediate noisy)
depth estimates to obtain reprojection-based conditioning to maintain
multi-view consistency. We train our model using large-scale synthetic dataset
Obajverse as well as the real-world CO3D dataset comprising of generic camera
viewpoints. We demonstrate that our approach can yield more accurate synthesis
compared to recent state-of-the-art, including distillation-based 3D inference
and prior multi-view generation methods. We also evaluate the geometry induced
by our multi-view depth prediction and find that it yields a more accurate
representation than other direct 3D inference approaches.

通过对多视角一致的 RGB-D 图像进行生成建模，我们提出了 MVD-Fusion：一种用于通过单视角推理 3D 的方法。

MVD-Fusion: 单视角 3D 通过深度一致的多视角生成

MVD-Fusion: Single-view 3D via Depth-consistent Multi-view Generation

We introduce a method for novel view synthesis given only a single
wide-baseline stereo image pair. In this challenging regime, 3D scene points
are regularly observed only once, requiring prior-based reconstruction of scene
geometry and appearance. We find that existing approaches to novel view
synthesis from sparse observations fail due to recovering incorrect 3D geometry
and due to the high cost of differentiable rendering that precludes their
scaling to large-scale training. We take a step towards resolving these
shortcomings by formulating a multi-view transformer encoder, proposing an
efficient, image-space epipolar line sampling scheme to assemble image features
for a target ray, and a lightweight cross-attention-based renderer. Our
contributions enable training of our method on a large-scale real-world dataset
of indoor and outdoor scenes. We demonstrate that our method learns powerful
multi-view geometry priors while reducing the rendering time. We conduct
extensive comparisons on held-out test scenes across two real-world datasets,
significantly outperforming prior work on novel view synthesis from sparse
image observations and achieving multi-view-consistent novel view synthesis.

介绍了一种用于单个广角立体图像对的新视角合成的方法，包括 3D 场景重构和外观变化的先验模型，提出了一种多视角变换编码器、图像上的极线采样方案和轻量级交叉注意力渲染器，通过大规模实际数据集的训练，证明了模型学习到了强大的多视角几何先验，大大缩短了渲染时间，并在两个实际数据集上得到了显著的优越性能。