Audio-driven talking face generation has garnered significant interest within
the domain of digital human research. Existing methods are encumbered by
intricate model architectures that are intricately dependent on each other,
complicating the process of re-editing image or video inputs. In this work, we
present ControlTalk, a talking face generation method to control face
expression deformation based on driven audio, which can construct the head pose
and facial expression including lip motion for both single image or sequential
video inputs in a unified manner. By utilizing a pre-trained video synthesis
renderer and proposing the lightweight adaptation, ControlTalk achieves precise
and naturalistic lip synchronization while enabling quantitative control over
mouth opening shape. Our experiments show that our method is superior to
state-of-the-art performance on widely used benchmarks, including HDTF and
MEAD. The parameterized adaptation demonstrates remarkable generalization
capabilities, effectively handling expression deformation across same-ID and
cross-ID scenarios, and extending its utility to out-of-domain portraits,
regardless of languages.

基于音频驱动的控制型对话生成系统，可根据音频控制面部表情变形，包括单幅图像或顺序视频输入，能够实现准确而自然的口型同步，并能够定量控制嘴巴张开的形状。在广泛使用的基准测试中，我们的实验表明我们的方法在性能上优于最先进的技术，可实现表情变形的跨身份和跨语种的处理，并扩展其在异域肖像中的使用。