Estimating 3D full-body pose from sparse sensor data is a pivotal technique
employed for the reconstruction of realistic human motions in Augmented Reality
and Virtual Reality. However, translating sparse sensor signals into
comprehensive human motion remains a challenge since the sparsely distributed
sensors in common VR systems fail to capture the motion of full human body. In
this paper, we use well-designed Body Pose Graph (BPG) to represent the human
body and translate the challenge into a prediction problem of graph missing
nodes. Then, we propose a novel full-body motion reconstruction framework based
on BPG. To establish BPG, nodes are initially endowed with features extracted
from sparse sensor signals. Features from identifiable joint nodes across
diverse sensors are amalgamated and processed from both temporal and spatial
perspectives. Temporal dynamics are captured using the Temporal Pyramid
Structure, while spatial relations in joint movements inform the spatial
attributes. The resultant features serve as the foundational elements of the
BPG nodes. To further refine the BPG, node features are updated through a graph
neural network that incorporates edge reflecting varying joint relations. Our
method's effectiveness is evidenced by the attained state-of-the-art
performance, particularly in lower body motion, outperforming other baseline
methods. Additionally, an ablation study validates the efficacy of each module
in our proposed framework.

从稀疏传感器数据中估计 3D 全身姿势是增强现实和虚拟现实中用于重建真实人体动作的关键技术之一。本文提出了一种基于 Body Pose Graph (BPG) 的新型全身动作重建框架，通过将人体表示为图形缺失节点预测问题来解决稀疏传感器信号到全身动作的转换挑战，在利用时间和空间信息处理了可识别关节点的特征后，通过图神经网络进一步改进 BPG 节点特征，实现了领先于其他基准方法的最先进性能。

基于图视角稀疏感知的全身动作重建

Full-Body Motion Reconstruction with Sparse Sensing from Graph  Perspective

Avatars are important to create interactive and immersive experiences in
virtual worlds. One challenge in animating these characters to mimic a user's
motion is that commercial AR/VR products consist only of a headset and
controllers, providing very limited sensor data of the user's pose. Another
challenge is that an avatar might have a different skeleton structure than a
human and the mapping between them is unclear. In this work we address both of
these challenges. We introduce a method to retarget motions in real-time from
sparse human sensor data to characters of various morphologies. Our method uses
reinforcement learning to train a policy to control characters in a physics
simulator. We only require human motion capture data for training, without
relying on artist-generated animations for each avatar. This allows us to use
large motion capture datasets to train general policies that can track unseen
users from real and sparse data in real-time. We demonstrate the feasibility of
our approach on three characters with different skeleton structure: a dinosaur,
a mouse-like creature and a human. We show that the avatar poses often match
the user surprisingly well, despite having no sensor information of the lower
body available. We discuss and ablate the important components in our
framework, specifically the kinematic retargeting step, the imitation, contact
and action reward as well as our asymmetric actor-critic observations. We
further explore the robustness of our method in a variety of settings including
unbalancing, dancing and sports motions.

本研究提出了一种利用强化学习、物理模拟器和稀疏传感器数据实时将人类运动重定向到具有不同骨架结构的角色的方法，并仅需要运用运动捕捉数据进行培训，同时探讨了该框架中的重要组件。