Learning 3D human motion from 2D inputs is a fundamental task in the realms
of computer vision and computer graphics. Many previous methods grapple with
this inherently ambiguous task by introducing motion priors into the learning
process. However, these approaches face difficulties in defining the complete
configurations of such priors or training a robust model. In this paper, we
present the Video-to-Motion Generator (VTM), which leverages motion priors
through cross-modal latent feature space alignment between 3D human motion and
2D inputs, namely videos and 2D keypoints. To reduce the complexity of modeling
motion priors, we model the motion data separately for the upper and lower body
parts. Additionally, we align the motion data with a scale-invariant virtual
skeleton to mitigate the interference of human skeleton variations to the
motion priors. Evaluated on AIST++, the VTM showcases state-of-the-art
performance in reconstructing 3D human motion from monocular videos. Notably,
our VTM exhibits the capabilities for generalization to unseen view angles and
in-the-wild videos.

该论文介绍了通过视频和 2D 关键点之间的跨模态潜在特征空间对三维人体运动和 2D 输入之间进行运动先验对齐的 Video-to-Motion Generator (VTM)，该方法通过将运动数据分别建模为上半身和下半身，以及使用尺度不变的虚拟骨架与运动数据对齐，展示了在重建单目视频中的三维人体运动方面具有领先水平的表现，尤其是在未见过的视角和野外视频中的泛化能力。