Estimating 3D full-body avatars from AR/VR devices is essential for creating
immersive experiences in AR/VR applications. This task is challenging due to
the limited input from Head Mounted Devices, which capture only sparse
observations from the head and hands. Predicting the full-body avatars,
particularly the lower body, from these sparse observations presents
significant difficulties. In this paper, we are inspired by the inherent
property of the kinematic tree defined in the Skinned Multi-Person Linear
(SMPL) model, where the upper body and lower body share only one common
ancestor node, bringing the potential of decoupled reconstruction. We propose a
stratified approach to decouple the conventional full-body avatar
reconstruction pipeline into two stages, with the reconstruction of the upper
body first and a subsequent reconstruction of the lower body conditioned on the
previous stage. To implement this straightforward idea, we leverage the latent
diffusion model as a powerful probabilistic generator, and train it to follow
the latent distribution of decoupled motions explored by a VQ-VAE
encoder-decoder model. Extensive experiments on AMASS mocap dataset demonstrate
our state-of-the-art performance in the reconstruction of full-body motions.

从 AR / VR 设备中估计 3D 全身化身对于创建 AR / VR 应用程序中的沉浸式体验至关重要。本文提出了一种分层方法，将传统的全身化身重建流程解耦为两个阶段，首先重建上半身，然后在先前阶段的基础上重建下半身。通过潜在扩散模型作为强大的概率生成器，并训练其遵循由 VQ-VAE 编码器 - 解码器模型探索的解耦动作的潜在分布，我们在 AMASS mocap 数据集上进行的广泛实验证明了我们在全身动作重建方面的最先进性能。

稀疏观测下的分层化角色生成

Stratified Avatar Generation from Sparse Observations

To bridge the physical and virtual worlds for rapidly developed VR/AR
applications, the ability to realistically drive 3D full-body avatars is of
great significance. Although real-time body tracking with only the head-mounted
displays (HMDs) and hand controllers is heavily under-constrained, a carefully
designed end-to-end neural network is of great potential to solve the problem
by learning from large-scale motion data. To this end, we propose a two-stage
framework that can obtain accurate and smooth full-body motions with the three
tracking signals of head and hands only. Our framework explicitly models the
joint-level features in the first stage and utilizes them as spatiotemporal
tokens for alternating spatial and temporal transformer blocks to capture
joint-level correlations in the second stage. Furthermore, we design a set of
loss terms to constrain the task of a high degree of freedom, such that we can
exploit the potential of our joint-level modeling. With extensive experiments
on the AMASS motion dataset and real-captured data, we validate the
effectiveness of our designs and show our proposed method can achieve more
accurate and smooth motion compared to existing approaches.

提出了一个两阶段的框架，仅利用头部和手部的追踪信号可以获取准确而平滑的全身动作，通过学习大规模动作数据解决实时身体追踪的问题，并通过关节级特征建模和损失函数设计提高精确度和平滑度。在 AMASS 运动数据集和真实捕获数据上的广泛实验验证了设计的有效性，并表明相比现有方法，我们的提出的方法可以实现更准确和平滑的运动。

基于关节级建模的稀疏观测的逼真全身跟踪

Realistic Full-Body Tracking from Sparse Observations via Joint-Level  Modeling

With the recent surge in popularity of AR/VR applications, realistic and
accurate control of 3D full-body avatars has become a highly demanded feature.
A particular challenge is that only a sparse tracking signal is available from
standalone HMDs (Head Mounted Devices), often limited to tracking the user's
head and wrists. While this signal is resourceful for reconstructing the upper
body motion, the lower body is not tracked and must be synthesized from the
limited information provided by the upper body joints. In this paper, we
present AGRoL, a novel conditional diffusion model specifically designed to
track full bodies given sparse upper-body tracking signals. Our model is based
on a simple multi-layer perceptron (MLP) architecture and a novel conditioning
scheme for motion data. It can predict accurate and smooth full-body motion,
particularly the challenging lower body movement. Unlike common diffusion
architectures, our compact architecture can run in real-time, making it
suitable for online body-tracking applications. We train and evaluate our model
on AMASS motion capture dataset, and demonstrate that our approach outperforms
state-of-the-art methods in generated motion accuracy and smoothness. We
further justify our design choices through extensive experiments and ablation
studies.

本研究提出一种基于条件扩散模型的新型解决方案 ——AGRoL，旨在根据稀疏的上肢追踪信号准确跟踪全身动作，通过在 AMASS 运动捕捉数据集上的训练和评估，证明我们的模型在预测和平滑全身运动方面优于现有的创新方法。