Text-guided domain adaption and generation of 3D-aware portraits find many
applications in various fields. However, due to the lack of training data and
the challenges in handling the high variety of geometry and appearance, the
existing methods for these tasks suffer from issues like inflexibility,
instability, and low fidelity. In this paper, we propose a novel framework
DiffusionGAN3D, which boosts text-guided 3D domain adaption and generation by
combining 3D GANs and diffusion priors. Specifically, we integrate the
pre-trained 3D generative models (e.g., EG3D) and text-to-image diffusion
models. The former provides a strong foundation for stable and high-quality
avatar generation from text. And the diffusion models in turn offer powerful
priors and guide the 3D generator finetuning with informative direction to
achieve flexible and efficient text-guided domain adaption. To enhance the
diversity in domain adaption and the generation capability in text-to-avatar,
we introduce the relative distance loss and case-specific learnable triplane
respectively. Besides, we design a progressive texture refinement module to
improve the texture quality for both tasks above. Extensive experiments
demonstrate that the proposed framework achieves excellent results in both
domain adaption and text-to-avatar tasks, outperforming existing methods in
terms of generation quality and efficiency. The project homepage is at
this https URL

我们提出了一个新的框架 DiffusionGAN3D，通过结合 3D GANs 和扩散先验来增强文本引导的 3D 领域适应和生成，该方法在领域适应和文本到头像任务中取得了优异的生成质量和效率结果。

DiffusionGAN3D: 结合 3D GANs 和 Diffusion 先验的文本引导的 3D 生成和领域适应

DiffusionGAN3D: Boosting Text-guided 3D Generation and Domain Adaption  by Combining 3D GANs and Diffusion Priors

Recently, significant advancements have been made in 3D generative models,
however training these models across diverse domains is challenging and
requires an huge amount of training data and knowledge of pose distribution.
Text-guided domain adaptation methods have allowed the generator to be adapted
to the target domains using text prompts, thereby obviating the need for
assembling numerous data. Recently, DATID-3D presents impressive quality of
samples in text-guided domain, preserving diversity in text by leveraging
text-to-image diffusion. However, adapting 3D generators to domains with
significant domain gaps from the source domain still remains challenging due to
issues in current text-to-image diffusion models as following: 1) shape-pose
trade-off in diffusion-based translation, 2) pose bias, and 3) instance bias in
the target domain, resulting in inferior 3D shapes, low text-image
correspondence, and low intra-domain diversity in the generated samples. To
address these issues, we propose a novel pipeline called PODIA-3D, which uses
pose-preserved text-to-image diffusion-based domain adaptation for 3D
generative models. We construct a pose-preserved text-to-image diffusion model
that allows the use of extremely high-level noise for significant domain
changes. We also propose specialized-to-general sampling strategies to improve
the details of the generated samples. Moreover, to overcome the instance bias,
we introduce a text-guided debiasing method that improves intra-domain
diversity. Consequently, our method successfully adapts 3D generators across
significant domain gaps. Our qualitative results and user study demonstrates
that our approach outperforms existing 3D text-guided domain adaptation methods
in terms of text-image correspondence, realism, diversity of rendered images,
and sense of depth of 3D shapes in the generated samples

提供一种基于姿态保持的语义引导扩散域适应方法，该方法采用高级噪声，改进采样策略，以及去偏差算法，成功地跨越了 3D 生成模型领域差距，提高了文本 - 图像对应性，真实性，生成图像的多样性，以及感知深度

PODIA-3D：使用保存姿态的文本到图像扩散进行跨大域 3D 生成模型域适应

PODIA-3D: Domain Adaptation of 3D Generative Model Across Large Domain  Gap Using Pose-Preserved Text-to-Image Diffusion

Recent 3D generative models have achieved remarkable performance in
synthesizing high resolution photorealistic images with view consistency and
detailed 3D shapes, but training them for diverse domains is challenging since
it requires massive training images and their camera distribution information.
Text-guided domain adaptation methods have shown impressive performance on
converting the 2D generative model on one domain into the models on other
domains with different styles by leveraging the CLIP (Contrastive
Language-Image Pre-training), rather than collecting massive datasets for those
domains. However, one drawback of them is that the sample diversity in the
original generative model is not well-preserved in the domain-adapted
generative models due to the deterministic nature of the CLIP text encoder.
Text-guided domain adaptation will be even more challenging for 3D generative
models not only because of catastrophic diversity loss, but also because of
inferior text-image correspondence and poor image quality. Here we propose
DATID-3D, a domain adaptation method tailored for 3D generative models using
text-to-image diffusion models that can synthesize diverse images per text
prompt without collecting additional images and camera information for the
target domain. Unlike 3D extensions of prior text-guided domain adaptation
methods, our novel pipeline was able to fine-tune the state-of-the-art 3D
generator of the source domain to synthesize high resolution, multi-view
consistent images in text-guided targeted domains without additional data,
outperforming the existing text-guided domain adaptation methods in diversity
and text-image correspondence. Furthermore, we propose and demonstrate diverse
3D image manipulations such as one-shot instance-selected adaptation and
single-view manipulated 3D reconstruction to fully enjoy diversity in text.

DATID-3D 是一种适用于 3D 生成模型的领域适应方法，采用文本到图像扩散模型，在不收集附加数据的情况下，将源领域的最先进 3D 生成器微调为文本引导的目标领域，从而实现高分辨率、多视角一致的图像合成，并提出并演示了多样的 3D 图像操作。