To bridge the physical and virtual worlds for rapidly developed VR/AR
applications, the ability to realistically drive 3D full-body avatars is of
great significance. Although real-time body tracking with only the head-mounted
displays (HMDs) and hand controllers is heavily under-constrained, a carefully
designed end-to-end neural network is of great potential to solve the problem
by learning from large-scale motion data. To this end, we propose a two-stage
framework that can obtain accurate and smooth full-body motions with the three
tracking signals of head and hands only. Our framework explicitly models the
joint-level features in the first stage and utilizes them as spatiotemporal
tokens for alternating spatial and temporal transformer blocks to capture
joint-level correlations in the second stage. Furthermore, we design a set of
loss terms to constrain the task of a high degree of freedom, such that we can
exploit the potential of our joint-level modeling. With extensive experiments
on the AMASS motion dataset and real-captured data, we validate the
effectiveness of our designs and show our proposed method can achieve more
accurate and smooth motion compared to existing approaches.

提出了一个两阶段的框架，仅利用头部和手部的追踪信号可以获取准确而平滑的全身动作，通过学习大规模动作数据解决实时身体追踪的问题，并通过关节级特征建模和损失函数设计提高精确度和平滑度。在 AMASS 运动数据集和真实捕获数据上的广泛实验验证了设计的有效性，并表明相比现有方法，我们的提出的方法可以实现更准确和平滑的运动。