Audio-driven portrait animation aims to synthesize portrait videos that are
conditioned by given audio. Animating high-fidelity and multimodal video
portraits has a variety of applications. Previous methods have attempted to
capture different motion modes and generate high-fidelity portrait videos by
training different models or sampling signals from given videos. However,
lacking correlation learning between lip-sync and other movements (e.g., head
pose/eye blinking) usually leads to unnatural results. In this paper, we
propose a unified system for multi-person, diverse, and high-fidelity talking
portrait generation. Our method contains three stages, i.e., 1) Mapping-Once
network with Dual Attentions (MODA) generates talking representation from given
audio. In MODA, we design a dual-attention module to encode accurate mouth
movements and diverse modalities. 2) Facial composer network generates dense
and detailed face landmarks, and 3) temporal-guided renderer syntheses stable
videos. Extensive evaluations demonstrate that the proposed system produces
more natural and realistic video portraits compared to previous methods.

本文提出了一个统一的系统，用于多人、多样性和高保真度的说话肖像生成，并通过对嘴部运动的准确编码和多样性的双注意力模块设计以及稳定视频合成呈现出更加自然和真实的视频肖像。