Recent works on text-to-3d generation show that using only 2D diffusion
supervision for 3D generation tends to produce results with inconsistent
appearances (e.g., faces on the back view) and inaccurate shapes (e.g., animals
with extra legs). Existing methods mainly address this issue by retraining
diffusion models with images rendered from 3D data to ensure multi-view
consistency while struggling to balance 2D generation quality with 3D
consistency. In this paper, we present a new framework Sculpt3D that equips the
current pipeline with explicit injection of 3D priors from retrieved reference
objects without re-training the 2D diffusion model. Specifically, we
demonstrate that high-quality and diverse 3D geometry can be guaranteed by
keypoints supervision through a sparse ray sampling approach. Moreover, to
ensure accurate appearances of different views, we further modulate the output
of the 2D diffusion model to the correct patterns of the template views without
altering the generated object's style. These two decoupled designs effectively
harness 3D information from reference objects to generate 3D objects while
preserving the generation quality of the 2D diffusion model. Extensive
experiments show our method can largely improve the multi-view consistency
while retaining fidelity and diversity. Our project page is available at:
this https URL

使用稀疏射线采样方法通过关键点监督，实现了在当前管道中显式注入来自检索参考对象的 3D 先验，以确保高质量和多样化的 3D 几何，同时保持 2D 扩散模型的生成质量和多视角一致性。