Head-mounted displays (HMDs) serve as indispensable devices for observing
extended reality (XR) environments and virtual content. However, HMDs present
an obstacle to external recording techniques as they block the upper face of
the user. This limitation significantly affects social XR applications,
specifically teleconferencing, where facial features and eye gaze information
play a vital role in creating an immersive user experience. In this study, we
propose a new network for expression-aware video inpainting for HMD removal
(EVI-HRnet) based on generative adversarial networks (GANs). Our model
effectively fills in missing information with regard to facial landmarks and a
single occlusion-free reference image of the user. The framework and its
components ensure the preservation of the user's identity across frames using
the reference frame. To further improve the level of realism of the inpainted
output, we introduce a novel facial expression recognition (FER) loss function
for emotion preservation. Our results demonstrate the remarkable capability of
the proposed framework to remove HMDs from facial videos while maintaining the
subject's facial expression and identity. Moreover, the outputs exhibit
temporal consistency along the inpainted frames. This lightweight framework
presents a practical approach for HMD occlusion removal, with the potential to
enhance various collaborative XR applications without the need for additional
hardware.

基于生成对抗网络（GANs）的表情感知视频修复（EVI-HRnet）网络通过使用用户的面部关键点和单一无遮挡参考图像，有效地填补头戴式显示器屏蔽的信息，并通过引入面部表情识别损失函数实现情感保留，从而实现了从面部视频中删除头戴式显示器同时保持主题的面部表情和身份，并在修复后的帧之间保持时间一致性。这种轻量级框架为头戴式显示器遮挡的去除提供了实用的方法，无需额外的硬件，有潜力增强各种协作的扩展性应用。

面部表情感知的 XR 应用中 HMD 移除的视频修复技术

Expression-aware video inpainting for HMD removal in XR applications

Facial motion tracking in head-mounted displays (HMD) has the potential to
enable immersive "face-to-face" interaction in a virtual environment. However,
current works on facial tracking are not suitable for unobtrusive augmented
reality (AR) glasses or do not have the ability to track arbitrary facial
movements. In this work, we demonstrate a novel system called SpecTracle that
tracks a user's facial motions using two wide-angle cameras mounted right next
to the visor of a Hololens. Avoiding the usage of cameras extended in front of
the face, our system greatly improves the feasibility to integrate full-face
tracking into a low-profile form factor. We also demonstrate that a neural
network-based model processing the wide-angle cameras can run in real-time at
24 frames per second (fps) on a mobile GPU and track independent facial
movement for different parts of the face with a user-independent model. Using a
short personalized calibration, the system improves its tracking performance by
42.3% compared to the user-independent model.

通过两个广角摄像头紧挨 Hololens 的遮光罩进行面部动作追踪，我们展示了一种名为 SpecTracle 的新型系统。避免使用前置摄像头，我们的系统大大提高了将全脸追踪融入低调外形的可行性。我们还证明了基于神经网络模型处理广角摄像头可以在移动 GPU 上实时运行，每秒 24 帧，并使用用户独立模型追踪面部的不同部位的独立动作。通过简短的个性化校准，与用户独立模型相比，该系统的追踪性能提高了 42.3%。

SpecTracle: 佩戴型无障碍视角摄像头面部动作追踪系统

SpecTracle: Wearable Facial Motion Tracking from Unobtrusive Peripheral  Cameras

With developments in computer graphics, hardware technology, perception
engineering, and human-computer interaction, virtual reality and virtual
environments are becoming more integrated into our daily lives. Head-mounted
displays, however, are still not used as frequently as other mobile devices
such as smart phones and watches. With increased usage of this technology and
the acclimation of humans to virtual application scenarios, it is possible that
in the near future an everyday virtual reality paradigm will be realized. When
considering the marriage of everyday virtual reality and head-mounted displays,
eye tracking is an emerging technology that helps to assess human behaviors in
a real time and non-intrusive way. Still, multiple aspects need to be
researched before these technologies become widely available in daily life.
Firstly, attention and cognition models in everyday scenarios should be
thoroughly understood. Secondly, as eyes are related to visual biometrics,
privacy preserving methodologies are necessary. Lastly, instead of studies or
applications utilizing limited human participants with relatively homogeneous
characteristics, protocols and use-cases for making such technology more
accessible should be essential. In this work, taking the aforementioned points
into account, a significant scientific push towards everyday virtual reality
has been completed with three main research contributions.

通过调查注意力和认知模型、隐私保护方法和协议，研究人员完成了 3 个主要研究贡献，旨在将 VR 和随身式头戴式设备推向日常生活。

通过眼动追踪实现日常虚拟现实

Towards Everyday Virtual Reality through Eye Tracking

The delivery and display of 360-degree videos on Head-Mounted Displays (HMDs)
presents many technical challenges. 360-degree videos are ultra high resolution
spherical videos, which contain an omnidirectional view of the scene. However
only a portion of this scene is displayed on the HMD. Moreover, HMD need to
respond in 10 ms to head movements, which prevents the server to send only the
displayed video part based on client feedback. To reduce the bandwidth waste,
while still providing an immersive experience, a viewport-adaptive 360-degree
video streaming system is proposed. The server prepares multiple video
representations, which differ not only by their bit-rate, but also by the
qualities of different scene regions. The client chooses a representation for
the next segment such that its bit-rate fits the available throughput and a
full quality region matches its viewing. We investigate the impact of various
spherical-to-plane projections and quality arrangements on the video quality
displayed to the user, showing that the cube map layout offers the best quality
for the given bit-rate budget. An evaluation with a dataset of users navigating
360-degree videos demonstrates that segments need to be short enough to enable
frequent view switches.

本文介绍了一种针对头戴显示器的多视点自适应 360 度视频流系统，实现了在提供沉浸式体验的同时降低带宽的浪费。研究表明铰接映射的布局是比较适合这种技术的。