We present AvatarPopUp, a method for fast, high quality 3D human avatar
generation from different input modalities, such as images and text prompts and
with control over the generated pose and shape. The common theme is the use of
diffusion-based image generation networks that are specialized for each
particular task, followed by a 3D lifting network. We purposefully decouple the
generation from the 3D modeling which allow us to leverage powerful image
synthesis priors, trained on billions of text-image pairs. We fine-tune latent
diffusion networks with additional image conditioning to solve tasks such as
image generation and back-view prediction, and to support qualitatively
different multiple 3D hypotheses. Our partial fine-tuning approach allows to
adapt the networks for each task without inducing catastrophic forgetting. In
our experiments, we demonstrate that our method produces accurate, high-quality
3D avatars with diverse appearance that respect the multimodal text, image, and
body control signals. Our approach can produce a 3D model in as few as 2
seconds, a four orders of magnitude speedup w.r.t. the vast majority of
existing methods, most of which solve only a subset of our tasks, and with
fewer controls, thus enabling applications that require the controlled 3D
generation of human avatars at scale. The project website can be found at
this https URL

通过 AvatarPopUp 方法，可以从不同输入模态生成快速且高质量的 3D 人体化身，具有对生成的姿势和形状的控制，使用基于扩散的图片生成网络和 3D 提升网络。该方法根据每个特定任务进行精细调优，能够在几秒内生成高质量的 3D 模型，远快于现有大多数方法，可以用于需要大规模控制 3D 生成的人体化身应用。

使用图像扩散模型进行即时三维人体化身生成

Instant 3D Human Avatar Generation using Image Diffusion Models

We introduce a novel framework for 3D human avatar generation and
personalization, leveraging text prompts to enhance user engagement and
customization. Central to our approach are key innovations aimed at overcoming
the challenges in photo-realistic avatar synthesis. Firstly, we utilize a
conditional Neural Radiance Fields (NeRF) model, trained on a large-scale
unannotated multi-view dataset, to create a versatile initial solution space
that accelerates and diversifies avatar generation. Secondly, we develop a
geometric prior, leveraging the capabilities of Text-to-Image Diffusion Models,
to ensure superior view invariance and enable direct optimization of avatar
geometry. These foundational ideas are complemented by our optimization
pipeline built on Variational Score Distillation (VSD), which mitigates texture
loss and over-saturation issues. As supported by our extensive experiments,
these strategies collectively enable the creation of custom avatars with
unparalleled visual quality and better adherence to input text prompts. You can
find more results and videos in our website:
this https URL

我们介绍了一个新颖的框架用于 3D 人体头像的生成和个性化，利用文本提示来提高用户参与度和定制性。通过克服面向照片真实感头像合成的挑战，我们的方法的核心创新在于应用条件神经辐射场（NeRF）模型，利用大规模多视角未注释数据集，创造出一个多功能的初始解空间，加速和多样化生成头像。此外，我们开发几何先验，利用文本到图像扩散模型的能力，确保视角不变性，并实现对头像几何的直接优化。这些基础思想结合我们基于变分分数蒸馏（VSD）的优化流程，有效减轻纹理损失和过饱和问题。通过广泛的实验验证，这些策略共同实现了具有无与伦比的视觉质量和更好的符合输入文本提示的定制头像的创建。您可以在我们的网站上找到更多的结果和视频。