One-shot 3D talking portrait generation aims to reconstruct a 3D avatar from
an unseen image, and then animate it with a reference video or audio to
generate a talking portrait video. The existing methods fail to simultaneously
achieve the goals of accurate 3D avatar reconstruction and stable talking face
animation. Besides, while the existing works mainly focus on synthesizing the
head part, it is also vital to generate natural torso and background segments
to obtain a realistic talking portrait video. To address these limitations, we
present Real3D-Potrait, a framework that (1) improves the one-shot 3D
reconstruction power with a large image-to-plane model that distills 3D prior
knowledge from a 3D face generative model; (2) facilitates accurate
motion-conditioned animation with an efficient motion adapter; (3) synthesizes
realistic video with natural torso movement and switchable background using a
head-torso-background super-resolution model; and (4) supports one-shot
audio-driven talking face generation with a generalizable audio-to-motion
model. Extensive experiments show that Real3D-Portrait generalizes well to
unseen identities and generates more realistic talking portrait videos compared
to previous methods.

Real3D-Potrait 是一种框架，通过使用大型图像到平面模型和高效的运动适配器，从而改进了一次性 3D 重建的能力，实现了精确的运动条件动画，并利用头 - 躯干 - 背景超分辨率模型生成具有自然躯干运动和可切换背景的逼真视频，同时支持一次性以音频驱动的说话脸生成。与以前的方法相比，广泛的实验证明 Real3D-Portrait 对于未见过的身份具有很好的泛化能力，并生成更逼真的说话肖像视频。

Real3D-Portrait: 一次合成逼真的 3D 语音肖像

Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesis

In this paper, we consider a novel problem of reconstructing a 3D human
avatar from multiple unconstrained frames, independent of assumptions on camera
calibration, capture space, and constrained actions. The problem should be
addressed by a framework that takes multiple unconstrained images as inputs,
and generates a shape-with-skinning avatar in the canonical space, finished in
one feed-forward pass. To this end, we present 3D Avatar Reconstruction in the
wild (ARwild), which first reconstructs the implicit skinning fields in a
multi-level manner, by which the image features from multiple images are
aligned and integrated to estimate a pixel-aligned implicit function that
represents the clothed shape. To enable the training and testing of the new
framework, we contribute a large-scale dataset, MVP-Human (Multi-View and
multi-Pose 3D Human), which contains 400 subjects, each of which has 15 scans
in different poses and 8-view images for each pose, providing 6,000 3D scans
and 48,000 images in total. Overall, benefits from the specific network
architecture and the diverse data, the trained model enables 3D avatar
reconstruction from unconstrained frames and achieves state-of-the-art
performance.

该论文提出了一个可以从多个非约束帧中重新构建 3D 人物肖像的框架，并设计了一个多级方法来重建隐式的蒙皮场，基于一次前馈生成一个形状与蒙皮的化身，为此贡献了一个包含 400 个主题的大型数据集，该训练模型可以从非约束帧中重建 3D_avatar，并达到最先进的性能水平。