Virtual Reality (VR) bares promise of social interactions that can feel more
immersive than other media. Key to this is the ability to accurately animate a
photorealistic avatar of one's likeness while wearing a VR headset. Although
high quality registration of person-specific avatars to headset-mounted camera
(HMC) images is possible in an offline setting, the performance of generic
realtime models are significantly degraded. Online registration is also
challenging due to oblique camera views and differences in modality. In this
work, we first show that the domain gap between the avatar and headset-camera
images is one of the primary sources of difficulty, where a transformer-based
architecture achieves high accuracy on domain-consistent data, but degrades
when the domain-gap is re-introduced. Building on this finding, we develop a
system design that decouples the problem into two parts: 1) an iterative
refinement module that takes in-domain inputs, and 2) a generic avatar-guided
image-to-image style transfer module that is conditioned on current estimation
of expression and head pose. These two modules reinforce each other, as image
style transfer becomes easier when close-to-ground-truth examples are shown,
and better domain-gap removal helps registration. Our system produces
high-quality results efficiently, obviating the need for costly offline
registration to generate personalized labels. We validate the accuracy and
efficiency of our approach through extensive experiments on a commodity
headset, demonstrating significant improvements over direct regression methods
as well as offline registration.

虚拟现实 (VR) 具有比其他媒体更具身临其境感的社交互动的潜力。关键在于能够在佩戴 VR 头显的同时准确地模拟一个逼真的个人化化身。本研究揭示了头显摄像头镜头与模型相差较大是导致实时模型性能下降的主要原因，并提出了一个系统设计，将问题分解为两部分：1) 一个迭代优化模块，处理同一领域的输入；2) 一个通用的以化身为导向的图像转换模块，以当前表情和头部姿态的估计为条件。这两个模块相互加强，通过展示接近真实的示例，使图像样式转换更容易，并改善领域差异移除。我们的系统高效地产生高质量的结果，不再需要昂贵的离线注册来生成个性化标签。通过在市售头显上进行大量实验证实了我们方法的准确性和效率，并在直接回归方法和离线注册上展现了显著的改进。