We present Multi-HMR, a strong single-shot model for multi-person 3D human
mesh recovery from a single RGB image. Predictions encompass the whole body,
i.e, including hands and facial expressions, using the SMPL-X parametric model
and spatial location in the camera coordinate system. Our model detects people
by predicting coarse 2D heatmaps of person centers, using features produced by
a standard Vision Transformer (ViT) backbone. It then predicts their whole-body
pose, shape and spatial location using a new cross-attention module called the
Human Prediction Head (HPH), with one query per detected center token,
attending to the entire set of features. As direct prediction of SMPL-X
parameters yields suboptimal results, we introduce CUFFS; the Close-Up Frames
of Full-Body Subjects dataset, containing humans close to the camera with
diverse hand poses. We show that incorporating this dataset into training
further enhances predictions, particularly for hands, enabling us to achieve
state-of-the-art performance. Multi-HMR also optionally accounts for camera
intrinsics, if available, by encoding camera ray directions for each image
token. This simple design achieves strong performance on whole-body and
body-only benchmarks simultaneously. We train models with various backbone
sizes and input resolutions. In particular, using a ViT-S backbone and
$448\times448$ input images already yields a fast and competitive model with
respect to state-of-the-art methods, while considering larger models and higher
resolutions further improve performance.

提出了 Multi-HMR，这是一个强大的单次拍摄模型，可以从单个 RGB 图像中恢复多人的 3D 人体网格。

多人全身人体网格恢复单次拍摄

Multi-HMR: Multi-Person Whole-Body Human Mesh Recovery in a Single Shot

Humans live within a 3D space and constantly interact with it to perform
tasks. Such interactions involve physical contact between surfaces that is
semantically meaningful. Our goal is to learn how humans interact with scenes
and leverage this to enable virtual characters to do the same. To that end, we
introduce a novel Human-Scene Interaction (HSI) model that encodes proximal
relationships, called POSA for "Pose with prOximitieS and contActs". The
representation of interaction is body-centric, which enables it to generalize
to new scenes. Specifically, POSA augments the SMPL-X parametric human body
model such that, for every mesh vertex, it encodes (a) the contact probability
with the scene surface and (b) the corresponding semantic scene label. We learn
POSA with a VAE conditioned on the SMPL-X vertices, and train on the PROX
dataset, which contains SMPL-X meshes of people interacting with 3D scenes, and
the corresponding scene semantics from the PROX-E dataset. We demonstrate the
value of POSA with two applications. First, we automatically place 3D scans of
people in scenes. We use a SMPL-X model fit to the scan as a proxy and then
find its most likely placement in 3D. POSA provides an effective representation
to search for "affordances" in the scene that match the likely contact
relationships for that pose. We perform a perceptual study that shows
significant improvement over the state of the art on this task. Second, we show
that POSA's learned representation of body-scene interaction supports monocular
human pose estimation that is consistent with a 3D scene, improving on the
state of the art. Our model and code are available for research purposes at
this https URL

本文提出了 POSA 模型，用来学习人体与场景的交互，包括接触概率和语义场景标签，并展示了其在 3D 人物自动放置和一致的单目人体姿态估计方面的改进。

学习人 - 场景交互生成 3D 场景

Populating 3D Scenes by Learning Human-Scene Interaction

A key step towards understanding human behavior is the prediction of 3D human
motion. Successful solutions have many applications in human tracking, HCI, and
graphics. Most previous work focuses on predicting a time series of future 3D
joint locations given a sequence 3D joints from the past. This Euclidean
formulation generally works better than predicting pose in terms of joint
rotations. Body joint locations, however, do not fully constrain 3D human pose,
leaving degrees of freedom undefined, making it hard to animate a realistic
human from only the joints. Note that the 3D joints can be viewed as a sparse
point cloud. Thus the problem of human motion prediction can be seen as point
cloud prediction. With this observation, we instead predict a sparse set of
locations on the body surface that correspond to motion capture markers. Given
such markers, we fit a parametric body model to recover the 3D shape and pose
of the person. These sparse surface markers also carry detailed information
about human movement that is not present in the joints, increasing the
naturalness of the predicted motions. Using the AMASS dataset, we train MOJO,
which is a novel variational autoencoder that generates motions from latent
frequencies. MOJO preserves the full temporal resolution of the input motion,
and sampling from the latent frequencies explicitly introduces high-frequency
components into the generated motion. We note that motion prediction methods
accumulate errors over time, resulting in joints or markers that diverge from
true human bodies. To address this, we fit SMPL-X to the predictions at each
time step, projecting the solution back onto the space of valid bodies. These
valid markers are then propagated in time. Experiments show that our method
produces state-of-the-art results and realistic 3D body animations. The code
for research purposes is at this https URL

本文提出了一种基于稀疏表面标记预测人体运动的方法，通过 MOJO 模型生成高频成分的动画，使用 SMPL-X 模型保证解决方案符合身体的真实形态，并取得了最先进的结果。

我们不仅仅是我们的关节：预测 3D 身体如何运动

We are More than Our Joints: Predicting how 3D Bodies Move

To facilitate the analysis of human actions, interactions and emotions, we
compute a 3D model of human body pose, hand pose, and facial expression from a
single monocular image. To achieve this, we use thousands of 3D scans to train
a new, unified, 3D model of the human body, SMPL-X, that extends SMPL with
fully articulated hands and an expressive face. Learning to regress the
parameters of SMPL-X directly from images is challenging without paired images
and 3D ground truth. Consequently, we follow the approach of SMPLify, which
estimates 2D features and then optimizes model parameters to fit the features.
We improve on SMPLify in several significant ways: (1) we detect 2D features
corresponding to the face, hands, and feet and fit the full SMPL-X model to
these; (2) we train a new neural network pose prior using a large MoCap
dataset; (3) we define a new interpenetration penalty that is both fast and
accurate; (4) we automatically detect gender and the appropriate body models
(male, female, or neutral); (5) our PyTorch implementation achieves a speedup
of more than 8x over Chumpy. We use the new method, SMPLify-X, to fit SMPL-X to
both controlled images and images in the wild. We evaluate 3D accuracy on a new
curated dataset comprising 100 images with pseudo ground-truth. This is a step
towards automatic expressive human capture from monocular RGB data. The models,
code, and data are available for research purposes at
this https URL

本研究提出了通过单目图像计算实现人体姿态、手部姿态和面部表情三维模型的方法，主要使用了一个新的、一致的、包含完全表达手和面部表情的人体模型 SMPL-X，可以对受控图像和自然环境图片进行三维模型匹配。