Head avatars animated by visual signals have gained popularity, particularly
in cross-driving synthesis where the driver differs from the animated
character, a challenging but highly practical approach. The recently presented
MegaPortraits model has demonstrated state-of-the-art results in this domain.
We conduct a deep examination and evaluation of this model, with a particular
focus on its latent space for facial expression descriptors, and uncover
several limitations with its ability to express intense face motions. To
address these limitations, we propose substantial changes in both training
pipeline and model architecture, to introduce our EMOPortraits model, where we:
Enhance the model's capability to faithfully support intense, asymmetric face
expressions, setting a new state-of-the-art result in the emotion transfer
task, surpassing previous methods in both metrics and quality.
Incorporate speech-driven mode to our model, achieving top-tier performance
in audio-driven facial animation, making it possible to drive source identity
through diverse modalities, including visual signal, audio, or a blend of both.
We propose a novel multi-view video dataset featuring a wide range of intense
and asymmetric facial expressions, filling the gap with absence of such data in
existing datasets.

我们对 MegaPortraits 模型进行了深入的检查和评估，重点关注其用于面部表情描述符的潜在空间，并发现了其表达强烈面部动作能力的几个局限。为解决这些限制，我们提出了针对训练流程和模型架构的重大改进，推出了我们的 EMOPortraits 模型。该模型增强了对于真实支持强烈非对称面部表情的能力，在情感转移任务中取得了新的技术水平，在指标和质量方面超越了以往的方法。此外，我们将语音驱动模式融入模型，实现了在音频驱动的面部动画中的顶级性能，使得通过不同的模态，包括视觉信号、音频或两者的混合驱动源标识成为可能。还提出了一个新颖的多视角视频数据集，包括广泛范围的强烈非对称面部表情，填补了现有数据集中这类数据的空白。

情绪增强的多模一次拍摄头像

EMOPortraits: Emotion-enhanced Multimodal One-shot Head Avatars

In this work, we advance the neural head avatar technology to the megapixel
resolution while focusing on the particularly challenging task of cross-driving
synthesis, i.e., when the appearance of the driving image is substantially
different from the animated source image. We propose a set of new neural
architectures and training methods that can leverage both medium-resolution
video data and high-resolution image data to achieve the desired levels of
rendered image quality and generalization to novel views and motion. We
demonstrate that suggested architectures and methods produce convincing
high-resolution neural avatars, outperforming the competitors in the
cross-driving scenario. Lastly, we show how a trained high-resolution neural
avatar model can be distilled into a lightweight student model which runs in
real-time and locks the identities of neural avatars to several dozens of
pre-defined source images. Real-time operation and identity lock are essential
for many practical applications head avatar systems.

本文提出了一种新的神经头像技术，可以以百万像素分辨率完成跨驱动综合任务，其基于新的神经结构和训练方法，可以同时利用中等分辨率视频数据和高分辨率图像数据来生成高质量的头像。通过实验证明，该技术在跨驱动方案中性能更好，并可在实时操作中锁定神经头像的身份，适用于实际应用场景。