We present MVD-Fusion: a method for single-view 3D inference via generative
modeling of multi-view-consistent RGB-D images. While recent methods pursuing
3D inference advocate learning novel-view generative models, these generations
are not 3D-consistent and require a distillation process to generate a 3D
output. We instead cast the task of 3D inference as directly generating
mutually-consistent multiple views and build on the insight that additionally
inferring depth can provide a mechanism for enforcing this consistency.
Specifically, we train a denoising diffusion model to generate multi-view RGB-D
images given a single RGB input image and leverage the (intermediate noisy)
depth estimates to obtain reprojection-based conditioning to maintain
multi-view consistency. We train our model using large-scale synthetic dataset
Obajverse as well as the real-world CO3D dataset comprising of generic camera
viewpoints. We demonstrate that our approach can yield more accurate synthesis
compared to recent state-of-the-art, including distillation-based 3D inference
and prior multi-view generation methods. We also evaluate the geometry induced
by our multi-view depth prediction and find that it yields a more accurate
representation than other direct 3D inference approaches.

通过对多视角一致的 RGB-D 图像进行生成建模，我们提出了 MVD-Fusion：一种用于通过单视角推理 3D 的方法。