We present SimXR, a method for controlling a simulated avatar from
information (headset pose and cameras) obtained from AR / VR headsets. Due to
the challenging viewpoint of head-mounted cameras, the human body is often
clipped out of view, making traditional image-based egocentric pose estimation
challenging. On the other hand, headset poses provide valuable information
about overall body motion, but lack fine-grained details about the hands and
feet. To synergize headset poses with cameras, we control a humanoid to track
headset movement while analyzing input images to decide body movement. When
body parts are seen, the movements of hands and feet will be guided by the
images; when unseen, the laws of physics guide the controller to generate
plausible motion. We design an end-to-end method that does not rely on any
intermediate representations and learns to directly map from images and headset
poses to humanoid control signals. To train our method, we also propose a
large-scale synthetic dataset created using camera configurations compatible
with a commercially available VR headset (Quest 2) and show promising results
on real-world captures. To demonstrate the applicability of our framework, we
also test it on an AR headset with a forward-facing camera.

SimXR 是一种从 AR/VR 头显获取的信息（头显姿态和摄像头）控制模拟化身的方法，通过分析输入图像决定人体运动并与头显姿态协同作用，从而实现手部和脚部的精细动作，同时利用物理定律生成合理的运动。