Talking face generation has a wide range of potential applications in the
field of virtual digital humans. However, rendering high-fidelity facial video
while ensuring lip synchronization is still a challenge for existing
audio-driven talking face generation approaches. To address this issue, we
propose HyperLips, a two-stage framework consisting of a hypernetwork for
controlling lips and a high-resolution decoder for rendering high-fidelity
faces.In the first stage, we construct a base face generation network that uses
the hypernetwork to control the encoding latent code of the visual face
information over audio. First, FaceEncoder is used to obtain latent code by
extracting features from the visual face information taken from the video
source containing the face frame.Then, HyperConv, which weighting parameters
are updated by HyperNet with the audio features as input, will modify the
latent code to synchronize the lip movement with the audio. Finally,
FaceDecoder will decode the modified and synchronized latent code into visual
face content. In the second stage, we obtain higher quality face videos through
a high-resolution decoder. To further improve the quality of face generation,
we trained a high-resolution decoder, HRDecoder, using face images and detected
sketches generated from the first stage as input.Extensive quantitative and
qualitative experiments show that our method outperforms state-of-the-art work
with more realistic, high-fidelity, and lip synchronization. Project page:
this https URL

通过使用 HyperLips，结合超网络和高分辨率解码器，本论文提出了一种解决高保真度脸部视频渲染及确保嘴唇同步的挑战的话语生成框架。