Accurate and reliable human motion reconstruction is crucial for creating
natural interactions of full-body avatars in Virtual Reality (VR) and
entertainment applications. As the Metaverse and social applications gain
popularity, users are seeking cost-effective solutions to create full-body
animations that are comparable in quality to those produced by commercial
motion capture systems. In order to provide affordable solutions, though, it is
important to minimize the number of sensors attached to the subject's body.
Unfortunately, reconstructing the full-body pose from sparse data is a heavily
under-determined problem. Some studies that use IMU sensors face challenges in
reconstructing the pose due to positional drift and ambiguity of the poses. In
recent years, some mainstream VR systems have released 6-degree-of-freedom
(6-DoF) tracking devices providing positional and rotational information.
Nevertheless, most solutions for reconstructing full-body poses rely on
traditional inverse kinematics (IK) solutions, which often produce
non-continuous and unnatural poses. In this article, we introduce SparsePoser,
a novel deep learning-based solution for reconstructing a full-body pose from a
reduced set of six tracking devices. Our system incorporates a
convolutional-based autoencoder that synthesizes high-quality continuous human
poses by learning the human motion manifold from motion capture data. Then, we
employ a learned IK component, made of multiple lightweight feed-forward neural
networks, to adjust the hands and feet toward the corresponding trackers. We
extensively evaluate our method on publicly available motion capture datasets
and with real-time live demos. We show that our method outperforms
state-of-the-art techniques using IMU sensors or 6-DoF tracking devices, and
can be used for users with different body dimensions and proportions.

SparsePoser 是一种深度学习为基础的解决方案，用于从一组减少的六个跟踪设备中重建全身姿势。该方法在公开可用的动作捕捉数据集上进行了广泛评估，并通过实时演示显示了优于使用 IMU 传感器或 6 自由度跟踪设备的最先进技术，可适用于具有不同身体尺寸和比例的用户。

SparsePoser: 实时全身运动重建从稀疏数据

SparsePoser: Real-time Full-body Motion Reconstruction from Sparse Data

We present a learning-based method for building driving-signal aware
full-body avatars. Our model is a conditional variational autoencoder that can
be animated with incomplete driving signals, such as human pose and facial
keypoints, and produces a high-quality representation of human geometry and
view-dependent appearance. The core intuition behind our method is that better
drivability and generalization can be achieved by disentangling the driving
signals and remaining generative factors, which are not available during
animation. To this end, we explicitly account for information deficiency in the
driving signal by introducing a latent space that exclusively captures the
remaining information, thus enabling the imputation of the missing factors
required during full-body animation, while remaining faithful to the driving
signal. We also propose a learnable localized compression for the driving
signal which promotes better generalization, and helps minimize the influence
of global chance-correlations often found in real datasets. For a given driving
signal, the resulting variational model produces a compact space of uncertainty
for missing factors that allows for an imputation strategy best suited to a
particular application. We demonstrate the efficacy of our approach on the
challenging problem of full-body animation for virtual telepresence with
driving signals acquired from minimal sensors placed in the environment and
mounted on a VR-headset.

本文提出了一种基于学习的方法，可以使用不完整的驾驶信号，如人体姿势和面部关键点，来建立全身化身，从而实现高质量的人类几何和视角相关的外观表现，同时通过分离驾驶信号和生成因素来提高可驾驶性和泛化性，还提出了一种可学习的局部压缩驾驶信号方法，以便更好地泛化，从而产生适合于特定应用的缺失因素插值策略，最后在利用来自环境中放置的最小传感器和固定在 VR 头显上的驾驶信号来实现虚拟遥感的全身动画问题上证明了本方法的有效性。