We present MVD-Fusion: a method for single-view 3D inference via generative
modeling of multi-view-consistent RGB-D images. While recent methods pursuing
3D inference advocate learning novel-view generative models, these generations
are not 3D-consistent and require a distillation process to generate a 3D
output. We instead cast the task of 3D inference as directly generating
mutually-consistent multiple views and build on the insight that additionally
inferring depth can provide a mechanism for enforcing this consistency.
Specifically, we train a denoising diffusion model to generate multi-view RGB-D
images given a single RGB input image and leverage the (intermediate noisy)
depth estimates to obtain reprojection-based conditioning to maintain
multi-view consistency. We train our model using large-scale synthetic dataset
Obajverse as well as the real-world CO3D dataset comprising of generic camera
viewpoints. We demonstrate that our approach can yield more accurate synthesis
compared to recent state-of-the-art, including distillation-based 3D inference
and prior multi-view generation methods. We also evaluate the geometry induced
by our multi-view depth prediction and find that it yields a more accurate
representation than other direct 3D inference approaches.

通过对多视角一致的 RGB-D 图像进行生成建模，我们提出了 MVD-Fusion：一种用于通过单视角推理 3D 的方法。

MVD-Fusion: 单视角 3D 通过深度一致的多视角生成

MVD-Fusion: Single-view 3D via Depth-consistent Multi-view Generation

We tackle the task of reconstructing hand-object interactions from short
video clips. Given an input video, our approach casts 3D inference as a
per-video optimization and recovers a neural 3D representation of the object
shape, as well as the time-varying motion and hand articulation. While the
input video naturally provides some multi-view cues to guide 3D inference,
these are insufficient on their own due to occlusions and limited viewpoint
variations. To obtain accurate 3D, we augment the multi-view signals with
generic data-driven priors to guide reconstruction. Specifically, we learn a
diffusion network to model the conditional distribution of (geometric)
renderings of objects conditioned on hand configuration and category label, and
leverage it as a prior to guide the novel-view renderings of the reconstructed
scene. We empirically evaluate our approach on egocentric videos across 6
object categories, and observe significant improvements over prior single-view
and multi-view methods. Finally, we demonstrate our system's ability to
reconstruct arbitrary clips from YouTube, showing both 1st and 3rd person
interactions.

我们通过对短视频剪辑进行手物体交互的重建任务。以输入视频为基础，我们的方法将 3D 推断视为每个视频的优化，并恢复物体形状的神经 3D 表示，以及时间变化的动作和手关节。尽管输入视频自然地提供了一些多视角线索来指导 3D 推断，但由于遮挡和有限的视角变化，这些线索本身并不足够。为了获得准确的 3D 结果，我们使用通用的数据驱动先验来引导重建过程中的多视角信号。具体来说，我们学习了一个扩散网络，来建模基于手配置和类别标签的对象（几何）渲染的条件分布，并利用它作为先验来指导重建场景的新视角渲染。我们在 6 个物体类别的自我中心视频上对我们的方法进行了实证评估，并观察到相较于先前的单视角和多视角方法有显著的改进。最后，我们展示了我们的系统能够从 YouTube 中重建任意剪辑，展示了第一人称和第三人称的交互。