We aim to edit the lip movements in talking video according to the given
speech while preserving the personal identity and visual details. The task can
be decomposed into two sub-problems: (1) speech-driven lip motion generation
and (2) visual appearance synthesis. Current solutions handle the two
sub-problems within a single generative model, resulting in a challenging
trade-off between lip-sync quality and visual details preservation. Instead, we
propose to disentangle the motion and appearance, and then generate them one by
one with a speech-to-motion diffusion model and a motion-conditioned appearance
generation model. However, there still remain challenges in each stage, such as
motion-aware identity preservation in (1) and visual details preservation in
(2). Therefore, to preserve personal identity, we adopt landmarks to represent
the motion, and further employ a landmark-based identity loss. To capture
motion-agnostic visual details, we use separate encoders to encode the lip,
non-lip appearance and motion, and then integrate them with a learned fusion
module. We train MyTalk on a large-scale and diverse dataset. Experiments show
that our method generalizes well to the unknown, even out-of-domain person, in
terms of both lip sync and visual detail preservation. We encourage the readers
to watch the videos on our project page (this https URL).

我们提出了一种通过语音生成唇部运动和生成视觉外观来编辑说话视频的方法，通过将唇部运动和视觉外观分离并分别生成，使用语音到运动扩散模型和运动条件下的外观生成模型。通过使用标记点来表示运动并采用基于标记点的身份损失，我们可以保留个人身份信息，并通过使用独立编码器对唇部、非唇部外观和运动进行编码，并通过学习融合模块将它们整合以捕捉运动无关的视觉细节。实验证明，我们的方法在唇部同步和视觉细节保持方面对未知的、甚至是不相关的人具有很好的泛化能力。