We propose a novel method for generating high-resolution videos of
talking-heads from speech audio and a single 'identity' image. Our method is
based on a convolutional neural network model that incorporates a pre-trained
StyleGAN generator. We model each frame as a point in the latent space of
StyleGAN so that a video corresponds to a trajectory through the latent space.
Training the network is in two stages. The first stage is to model trajectories
in the latent space conditioned on speech utterances. To do this, we use an
existing encoder to invert the generator, mapping from each video frame into
the latent space. We train a recurrent neural network to map from speech
utterances to displacements in the latent space of the image generator. These
displacements are relative to the back-projection into the latent space of an
identity image chosen from the individuals depicted in the training dataset. In
the second stage, we improve the visual quality of the generated videos by
tuning the image generator on a single image or a short video of any chosen
identity. We evaluate our model on standard measures (PSNR, SSIM, FID and LMD)
and show that it significantly outperforms recent state-of-the-art methods on
one of two commonly used datasets and gives comparable performance on the
other. Finally, we report on ablation experiments that validate the components
of the model. The code and videos from experiments can be found at
this https URL

本文提出了一种基于卷积神经网络模型和预先训练的 StyleGAN 生成器的新方法，用于从语音音频和单个 ' 身份 ' 图像生成高分辨率的说话者视频。模型首先使用一个现有的编码器将每个视频帧映射到潜在空间中，然后从语音语句到图像生成器的潜在空间中的位移映射。最终，评估结果表明本文方法在标准指标上优于最新的先进方法，并在另一个常用数据集上获得可比较的性能。