The task of face reenactment is to transfer the head motion and facial
expressions from a driving video to the appearance of a source image, which may
be of a different person (cross-reenactment). Most existing methods are
CNN-based and estimate optical flow from the source image to the current
driving frame, which is then inpainted and refined to produce the output
animation. We propose a transformer-based encoder for computing a set-latent
representation of the source image(s). We then predict the output color of a
query pixel using a transformer-based decoder, which is conditioned with
keypoints and a facial expression vector extracted from the driving frame.
Latent representations of the source person are learned in a self-supervised
manner that factorize their appearance, head pose, and facial expressions.
Thus, they are perfectly suited for cross-reenactment. In contrast to most
related work, our method naturally extends to multiple source images and can
thus adapt to person-specific facial dynamics. We also propose data
augmentation and regularization schemes that are necessary to prevent
overfitting and support generalizability of the learned representations. We
evaluated our approach in a randomized user study. The results indicate
superior performance compared to the state-of-the-art in terms of motion
transfer quality and temporal consistency.

人脸再现的任务是将视频中的头部运动和面部表情转移至源图像的外观，可以是不同的人（跨再现）。我们提出了一种基于 Transformer 编码器的方法来计算源图像的一组潜在表示。然后，我们使用基于 Transformer 的解码器预测查询像素的输出颜色，该解码器受到来自驾驶帧的关键点和面部表情向量的条件影响。我们以自监督的方式学习源人物的潜在表示，这种表示分解了外观、头部角度和面部表情，非常适合于跨再现。与大多数相关工作相比，我们的方法自然地扩展到多个源图像，并且可以适应个人特定的面部动态。我们还提出了必要的数据增强和正则化方案，以防止过拟合并支持学习表示的泛化能力。我们在一个随机用户研究中评估了我们的方法，结果表明在运动传输质量和时间一致性方面与最先进方法相比具有卓越性能。