We show, for the first time, that neural networks trained only on synthetic
data achieve state-of-the-art accuracy on the problem of 3D human pose and
shape (HPS) estimation from real images. Previous synthetic datasets have been
small, unrealistic, or lacked realistic clothing. Achieving sufficient realism
is non-trivial and we show how to do this for full bodies in motion.
Specifically, our BEDLAM dataset contains monocular RGB videos with
ground-truth 3D bodies in SMPL-X format. It includes a diversity of body
shapes, motions, skin tones, hair, and clothing. The clothing is realistically
simulated on the moving bodies using commercial clothing physics simulation. We
render varying numbers of people in realistic scenes with varied lighting and
camera motions. We then train various HPS regressors using BEDLAM and achieve
state-of-the-art accuracy on real-image benchmarks despite training with
synthetic data. We use BEDLAM to gain insights into what model design choices
are important for accuracy. With good synthetic training data, we find that a
basic method like HMR approaches the accuracy of the current SOTA method
(CLIFF). BEDLAM is useful for a variety of tasks and all images, ground truth
bodies, 3D clothing, support code, and more are available for research
purposes. Additionally, we provide detailed information about our synthetic
data generation pipeline, enabling others to generate their own datasets. See
the project page: this https URL

通过对合成数据的训练，我们首次展示了神经网络在从实际图像中估计三维人体姿态和形状（HPS）问题上实现了最先进的精度。

BEDLAM: 一个合成数据集，展示身体具有详细的逼真动态

BEDLAM: A Synthetic Dataset of Bodies Exhibiting Detailed Lifelike  Animated Motion

Top-down methods dominate the field of 3D human pose and shape estimation,
because they are decoupled from human detection and allow researchers to focus
on the core problem. However, cropping, their first step, discards the location
information from the very beginning, which makes themselves unable to
accurately predict the global rotation in the original camera coordinate
system. To address this problem, we propose to Carry Location Information in
Full Frames (CLIFF) into this task. Specifically, we feed more holistic
features to CLIFF by concatenating the cropped-image feature with its bounding
box information. We calculate the 2D reprojection loss with a broader view of
the full frame, taking a projection process similar to that of the person
projected in the image. Fed and supervised by global-location-aware
information, CLIFF directly predicts the global rotation along with more
accurate articulated poses. Besides, we propose a pseudo-ground-truth annotator
based on CLIFF, which provides high-quality 3D annotations for in-the-wild 2D
datasets and offers crucial full supervision for regression-based methods.
Extensive experiments on popular benchmarks show that CLIFF outperforms prior
arts by a significant margin, and reaches the first place on the AGORA
leaderboard (the SMPL-Algorithms track). The code and data are available at
this https URL

本文提出一种新方法，将完整画面中的位置信息带入剪裁图像的特征中，以解决在 3D 人体姿态和形状估计中全局旋转不准确的问题，并提出基于 CLIFF 的伪基础真值标注器，能为野外 2D 数据集提供高质量的 3D 标注和对基于回归的方法提供关键的全面监督。

CLIFF：在完整帧中携带位置信息以进行人体姿态和形状估计

CLIFF: Carrying Location Information in Full Frames into Human Pose and  Shape Estimation

We propose a novel sparse constrained formulation and from it derive a
real-time optimization method for 3D human pose and shape estimation. Our
optimization method, SCOPE (Sparse Constrained Optimization for 3D human Pose
and shapE estimation), is orders of magnitude faster (avg. 4 ms convergence)
than existing optimization methods, while being mathematically equivalent to
their dense unconstrained formulation under mild assumptions. We achieve this
by exploiting the underlying sparsity and constraints of our formulation to
efficiently compute the Gauss-Newton direction. We show that this computation
scales linearly with the number of joints and measurements of a complex 3D
human model, in contrast to prior work where it scales cubically due to their
dense unconstrained formulation. Based on our optimization method, we present a
real-time motion capture framework that estimates 3D human poses and shapes
from a single image at over 30 FPS. In benchmarks against state-of-the-art
methods on multiple public datasets, our framework outperforms other
optimization methods and achieves competitive accuracy against regression
methods. Project page with code and videos:
this https URL

该研究提出了一种新颖的基于稀疏约束的 3D 人体姿态和形状估计方法，通过高效计算 Gauss-Newton 方向并利用优化问题的稀疏约束性质，将优化时间缩短到 4ms 左右，并成功地将其应用于单张图像的实时运动捕捉领域。