We propose FaceVR, a novel image-based method that enables video
teleconferencing in VR based on self-reenactment. State-of-the-art face
tracking methods in the VR context are focused on the animation of rigged 3d
avatars. While they achieve good tracking performance the results look
cartoonish and not real. In contrast to these model-based approaches, FaceVR
enables VR teleconferencing using an image-based technique that results in
nearly photo-realistic outputs. The key component of FaceVR is a robust
algorithm to perform real-time facial motion capture of an actor who is wearing
a head-mounted display (HMD), as well as a new data-driven approach for eye
tracking from monocular videos. Based on reenactment of a prerecorded stereo
video of the person without the HMD, FaceVR incorporates photo-realistic
re-rendering in real time, thus allowing artificial modifications of face and
eye appearances. For instance, we can alter facial expressions or change gaze
directions in the prerecorded target video. In a live setup, we apply these
newly-introduced algorithmic components.

提出了一种基于图像的新方法 FaceVR，该方法可基于自我再现实现 VR 视频会议，通过实时面部动作捕捉算法和单目视频的新数据驱动方法，FaceVR 具备近乎逼真的成像效果和改变面部表情和目光方向的功能。