We introduce SkelFormer, a novel markerless motion capture pipeline for
multi-view human pose and shape estimation. Our method first uses off-the-shelf
2D keypoint estimators, pre-trained on large-scale in-the-wild data, to obtain
3D joint positions. Next, we design a regression-based inverse-kinematic
skeletal transformer that maps the joint positions to pose and shape
representations from heavily noisy observations. This module integrates prior
knowledge about pose space and infers the full pose state at runtime.
Separating the 3D keypoint detection and inverse-kinematic problems, along with
the expressive representations learned by our skeletal transformer, enhance the
generalization of our method to unseen noisy data. We evaluate our method on
three public datasets in both in-distribution and out-of-distribution settings
using three datasets, and observe strong performance with respect to prior
works. Moreover, ablation experiments demonstrate the impact of each of the
modules of our architecture. Finally, we study the performance of our method in
dealing with noise and heavy occlusions and find considerable robustness with
respect to other solutions.

为了多视角人体姿势和形状估计，我们介绍了 SkelFormer，一种新颖的无标记动作捕捉方法。我们的方法首先使用现成的在野外大规模训练的 2D 关键点估计器来获取 3D 关节位置。接下来，我们设计了一个基于回归的逆运动学骨骼变换器，将关节位置从嘈杂的观测映射到姿势和形状表示。该模块结合了关于姿势空间的先验知识，并在运行时推断完整的姿势状态。通过将 3D 关键点检测和逆运动学问题分离，并利用我们骨骼变换器学到的表达能力，增强了我们的方法对未见过的嘈杂数据的泛化能力。我们在三个公共数据集上进行实验评估，在分布内和分布外的设置下，对之前的工作表现出良好的性能。此外，消融实验展示了我们架构的每个模块的影响。最后，我们研究了我们的方法在处理噪声和重叠方面的性能，并发现与其他解决方案相比具有相当的鲁棒性。