We propose 360{\deg} Volumetric Portrait (3VP) Avatar, a novel method for
reconstructing 360{\deg} photo-realistic portrait avatars of human subjects
solely based on monocular video inputs. State-of-the-art monocular avatar
reconstruction methods rely on stable facial performance capturing. However,
the common usage of 3DMM-based facial tracking has its limits; side-views can
hardly be captured and it fails, especially, for back-views, as required inputs
like facial landmarks or human parsing masks are missing. This results in
incomplete avatar reconstructions that only cover the frontal hemisphere. In
contrast to this, we propose a template-based tracking of the torso, head and
facial expressions which allows us to cover the appearance of a human subject
from all sides. Thus, given a sequence of a subject that is rotating in front
of a single camera, we train a neural volumetric representation based on neural
radiance fields. A key challenge to construct this representation is the
modeling of appearance changes, especially, in the mouth region (i.e., lips and
teeth). We, therefore, propose a deformation-field-based blend basis which
allows us to interpolate between different appearance states. We evaluate our
approach on captured real-world data and compare against state-of-the-art
monocular reconstruction methods. In contrast to those, our method is the first
monocular technique that reconstructs an entire 360{\deg} avatar.

基于单目视频输入，本研究提出了一种名为 360° 体积肖像（3VP）头像的新方法，该方法用于重建 360° 全景真实头像，以模板为基础跟踪躯干、头部和面部表情，训练基于神经辐射场的神经体积表示，解决了外观变化模型中特别是嘴唇和牙齿区域的挑战，评估了实际采集的数据并与最先进的单目重建方法进行了比较。

360° 体感肖像化头像

360° Volumetric Portrait Avatar

While high fidelity and efficiency are central to the creation of digital
head avatars, recent methods relying on 2D or 3D generative models often
experience limitations such as shape distortion, expression inaccuracy, and
identity flickering. Additionally, existing one-shot inversion techniques fail
to fully leverage multiple input images for detailed feature extraction. We
propose a novel framework, \textbf{Incremental 3D GAN Inversion}, that enhances
avatar reconstruction performance using an algorithm designed to increase the
fidelity from multiple frames, resulting in improved reconstruction quality
proportional to frame count. Our method introduces a unique animatable 3D GAN
prior with two crucial modifications for enhanced expression controllability
alongside an innovative neural texture encoder that categorizes texture feature
spaces based on UV parameterization. Differentiating from traditional
techniques, our architecture emphasizes pixel-aligned image-to-image
translation, mitigating the need to learn correspondences between observation
and canonical spaces. Furthermore, we incorporate ConvGRU-based recurrent
networks for temporal data aggregation from multiple frames, boosting geometry
and texture detail reconstruction. The proposed paradigm demonstrates
state-of-the-art performance on one-shot and few-shot avatar animation tasks.

提出了一种新颖的框架，增量 3D GAN 逆转，通过增加逆转质量随帧数增加来提高数字头像重建性能，引入了一个带有两个关键修改的唯一可动画的 3D GAN 先验，以及一种基于 UV 参数化对纹理特征空间进行分类的创新神经纹理编码器。