In this paper, we present StyleLipSync, a style-based personalized lip-sync
video generative model that can generate identity-agnostic lip-synchronizing
video from arbitrary audio. To generate a video of arbitrary identities, we
leverage expressive lip prior from the semantically rich latent space of a
pre-trained StyleGAN, where we can also design a video consistency with a
linear transformation. In contrast to the previous lip-sync methods, we
introduce pose-aware masking that dynamically locates the mask to improve the
naturalness over frames by utilizing a 3D parametric mesh predictor frame by
frame. Moreover, we propose a few-shot lip-sync adaptation method for an
arbitrary person by introducing a sync regularizer that preserves lips-sync
generalization while enhancing the person-specific visual information.
Extensive experiments demonstrate that our model can generate accurate lip-sync
videos even with the zero-shot setting and enhance characteristics of an unseen
face using a few seconds of target video through the proposed adaptation
method. Please refer to our project page.

本文介绍了 StyleLipSync，这是一个基于风格的个性化唇形同步视频生成模型，可从任意音频生成与身份无关的唇形同步视频。模型通过在预先训练的 StyleGAN 的语义丰富潜空间中利用表达性嘴唇先验来生成具有任意身份的视频。与之前的唇形同步方法相比，我们引入了姿态感知掩蔽，通过逐帧使用 3D 参数网格预测器动态定位掩蔽，以提高过帧自然性。此外，我们还提出了一种少量样本的唇形同步适应方法，引入同步正则器来保持唇形同步的泛化性，同时增强个人特定的视觉信息。广泛的实验证明，我们的模型可以生成准确的唇形同步视频，甚至可以在零样本设置下通过拟合少量秒数的目标视频增强看不见的面部特征。