Different people speak with diverse personalized speaking styles. Although
existing one-shot talking head methods have made significant progress in lip
sync, natural facial expressions, and stable head motions, they still cannot
generate diverse speaking styles in the final talking head videos. To tackle
this problem, we propose a one-shot style-controllable talking face generation
framework. In a nutshell, we aim to attain a speaking style from an arbitrary
reference speaking video and then drive the one-shot portrait to speak with the
reference speaking style and another piece of audio. Specifically, we first
develop a style encoder to extract dynamic facial motion patterns of a style
reference video and then encode them into a style code. Afterward, we introduce
a style-controllable decoder to synthesize stylized facial animations from the
speech content and style code. In order to integrate the reference speaking
style into generated videos, we design a style-aware adaptive transformer,
which enables the encoded style code to adjust the weights of the feed-forward
layers accordingly. Thanks to the style-aware adaptation mechanism, the
reference speaking style can be better embedded into synthesized videos during
decoding. Extensive experiments demonstrate that our method is capable of
generating talking head videos with diverse speaking styles from only one
portrait image and an audio clip while achieving authentic visual effects.
Project Page: this https URL

提出了一种一次性风格可控的说话人脸生成框架，可以从任意参考说话视频中获得一种说话风格，并将一次性画像驱动为使用参考说话风格和其他音频进行说话。