Despite recent advances in syncing lip movements with any audio waves,
current methods still struggle to balance generation quality and the model's
generalization ability. Previous studies either require long-term data for
training or produce a similar movement pattern on all subjects with low
quality. In this paper, we propose StyleSync, an effective framework that
enables high-fidelity lip synchronization. We identify that a style-based
generator would sufficiently enable such a charming property on both one-shot
and few-shot scenarios. Specifically, we design a mask-guided spatial
information encoding module that preserves the details of the given face. The
mouth shapes are accurately modified by audio through modulated convolutions.
Moreover, our design also enables personalized lip-sync by introducing style
space and generator refinement on only limited frames. Thus the identity and
talking style of a target person could be accurately preserved. Extensive
experiments demonstrate the effectiveness of our method in producing
high-fidelity results on a variety of scenes. Resources can be found at
this https URL

该研究提出了 StyleSync，一种有效的框架，通过引入风格空间和特征重构，利用音频的调制卷积来准确地修改嘴的形状，使得目标人物的身份和说话风格能够被准确地保留，从而实现高保真的唇部同步。