While recent works have achieved great success on one-shot 3D common object
generation, high quality and fidelity 3D head generation from a single image
remains a great challenge. Previous text-based methods for generating 3D heads
were limited by text descriptions and image-based methods struggled to produce
high-quality head geometry. To handle this challenging problem, we propose a
novel framework, Portrait3D, to generate high-quality 3D heads while preserving
their identities. Our work incorporates the identity information of the
portrait image into three parts: 1) geometry initialization, 2) geometry
sculpting, and 3) texture generation stages. Given a reference portrait image,
we first align the identity features with text features to realize ID-aware
guidance enhancement, which contains the control signals representing the face
information. We then use the canny map, ID features of the portrait image, and
a pre-trained text-to-normal/depth diffusion model to generate ID-aware
geometry supervision, and 3D-GAN inversion is employed to generate ID-aware
geometry initialization. Furthermore, with the ability to inject identity
information into 3D head generation, we use ID-aware guidance to calculate
ID-aware Score Distillation (ISD) for geometry sculpting. For texture
generation, we adopt the ID Consistent Texture Inpainting and Refinement which
progressively expands the view for texture inpainting to obtain an
initialization UV texture map. We then use the id-aware guidance to provide
image-level supervision for noisy multi-view images to obtain a refined texture
map. Extensive experiments demonstrate that we can generate high-quality 3D
heads with accurate geometry and texture from single in-the-wild portrait
images. The project page is at this https URL

提出了 Portrait3D 框架，通过将肖像图像的身份信息整合到几何初始化、几何塑造和纹理生成阶段中，实现了从单幅肖像图像生成高质量 3D 头像的方法。

Portrait3D：基于单张野外人像图像的 3D 头部生成

Portrait3D: 3D Head Generation from Single In-the-wild Portrait Image

We present DreamCraft3D, a hierarchical 3D content generation method that
produces high-fidelity and coherent 3D objects. We tackle the problem by
leveraging a 2D reference image to guide the stages of geometry sculpting and
texture boosting. A central focus of this work is to address the consistency
issue that existing works encounter. To sculpt geometries that render
coherently, we perform score distillation sampling via a view-dependent
diffusion model. This 3D prior, alongside several training strategies,
prioritizes the geometry consistency but compromises the texture fidelity. We
further propose Bootstrapped Score Distillation to specifically boost the
texture. We train a personalized diffusion model, Dreambooth, on the augmented
renderings of the scene, imbuing it with 3D knowledge of the scene being
optimized. The score distillation from this 3D-aware diffusion prior provides
view-consistent guidance for the scene. Notably, through an alternating
optimization of the diffusion prior and 3D scene representation, we achieve
mutually reinforcing improvements: the optimized 3D scene aids in training the
scene-specific diffusion model, which offers increasingly view-consistent
guidance for 3D optimization. The optimization is thus bootstrapped and leads
to substantial texture boosting. With tailored 3D priors throughout the
hierarchical generation, DreamCraft3D generates coherent 3D objects with
photorealistic renderings, advancing the state-of-the-art in 3D content
generation. Code available at this https URL

DreamCraft3D 是一个层次化的 3D 内容生成方法，通过使用 2D 参考图像来引导几何雕刻和纹理增强的阶段，解决了现有方法遇到的一致性问题，使用得分蒸馏采样和视角相关的扩散模型来生成一致的几何体，并通过训练个性化的扩散模型来提高纹理质量，最终生成具有逼真渲染效果的连贯的 3D 对象。