We present DiffPortrait3D, a conditional diffusion model that is capable of
synthesizing 3D-consistent photo-realistic novel views from as few as a single
in-the-wild portrait. Specifically, given a single RGB input, we aim to
synthesize plausible but consistent facial details rendered from novel camera
views with retained both identity and facial expression. In lieu of
time-consuming optimization and fine-tuning, our zero-shot method generalizes
well to arbitrary face portraits with unposed camera views, extreme facial
expressions, and diverse artistic depictions. At its core, we leverage the
generative prior of 2D diffusion models pre-trained on large-scale image
datasets as our rendering backbone, while the denoising is guided with
disentangled attentive control of appearance and camera pose. To achieve this,
we first inject the appearance context from the reference image into the
self-attention layers of the frozen UNets. The rendering view is then
manipulated with a novel conditional control module that interprets the camera
pose by watching a condition image of a crossed subject from the same view.
Furthermore, we insert a trainable cross-view attention module to enhance view
consistency, which is further strengthened with a novel 3D-aware noise
generation process during inference. We demonstrate state-of-the-art results
both qualitatively and quantitatively on our challenging in-the-wild and
multi-view benchmarks.

我们提出了 DiffPortrait3D，这是一种条件扩散模型，能够从一个野外肖像合成一致的 3D 真实场景。我们利用大规模图像数据预训练的 2D 扩散模型作为渲染骨干，通过外貌和相机姿势的解耦集中控制对图像进行去噪。我们进一步引入可训练的跨视角注意力模块以增强视角一致性，并在推断过程中实施一种新颖的 3D 感知噪声生成过程。在野外和多视角基准测试上，我们展示了优于现有方法的结果。