Editing and manipulating facial features in videos is an interesting and
important field of research with a plethora of applications, ranging from movie
post-production and visual effects to realistic avatars for video games and
virtual assistants. Our method supports semantic video manipulation based on
neural rendering and 3D-based facial expression modelling. We focus on
interactive manipulation of the videos by altering and controlling the facial
expressions, achieving promising photorealistic results. The proposed method is
based on a disentangled representation and estimation of the 3D facial shape
and activity, providing the user with intuitive and easy-to-use control of the
facial expressions in the input video. We also introduce a user-friendly,
interactive AI tool that processes human-readable semantic labels about the
desired expression manipulations in specific parts of the input video and
synthesizes photorealistic manipulated videos. We achieve that by mapping the
emotion labels to points on the Valence-Arousal space (where Valence quantifies
how positive or negative is an emotion and Arousal quantifies the power of the
emotion activation), which in turn are mapped to disentangled 3D facial
expressions through an especially-designed and trained expression decoder
network. The paper presents detailed qualitative and quantitative experiments,
which demonstrate the effectiveness of our system and the promising results it
achieves.

本文提出了一种基于神经渲染和基于 3D 面部表情建模的方法，支持语义视频操作，可以通过改变和控制面部表情实现交互式操作，实现了有希望的逼真效果，并且引入了用户友好的交互 AI 工具，可以处理关于特定部分的输入视频中期望的表情操作的可读的语义标签，并合成逼真的操作视频。