Existing single image-to-3D creation methods typically involve a two-stage
process, first generating multi-view images, and then using these images for 3D
reconstruction. However, training these two stages separately leads to
significant data bias in the inference phase, thus affecting the quality of
reconstructed results. We introduce a unified 3D generation framework, named
Ouroboros3D, which integrates diffusion-based multi-view image generation and
3D reconstruction into a recursive diffusion process. In our framework, these
two modules are jointly trained through a self-conditioning mechanism, allowing
them to adapt to each other's characteristics for robust inference. During the
multi-view denoising process, the multi-view diffusion model uses the 3D-aware
maps rendered by the reconstruction module at the previous timestep as
additional conditions. The recursive diffusion framework with 3D-aware feedback
unites the entire process and improves geometric consistency.Experiments show
that our framework outperforms separation of these two stages and existing
methods that combine them at the inference phase. Project page:
this https URL

现有的单图像到 3D 生成方法通常涉及两个阶段的过程，首先生成多视图图像，然后使用这些图像进行 3D 重建。然而，分别训练这两个阶段会导致推理阶段的数据偏差，从而影响重建结果的质量。我们引入了一个统一的 3D 生成框架，命名为 Ouroboros3D，它将基于扩散的多视图图像生成和 3D 重建集成到递归扩散过程中。在我们的框架中，通过自我条件机制联合训练这两个模块，使它们能够适应彼此的特征以进行稳健的推理。在多视图去噪过程中，多视图扩散模型使用由重建模块在上一时间步骤渲染的具有 3D 感知的地图作为附加条件。具有 3D 感知反馈的递归扩散框架统一了整个过程并改进了几何一致性。实验证明，我们的框架优于将这两个阶段分离和将它们合并在推理阶段的现有方法。