We present a learning-based model to infer the personalized 3D shape of
people from a few frames (1-8) of a monocular video in which the person is
moving, in less than 10 seconds with a reconstruction accuracy of 5mm. Our
model learns to predict the parameters of a statistical body model and instance
displacements that add clothing and hair to the shape. The model achieves fast
and accurate predictions based on two key design choices. First, by predicting
shape in a canonical T-pose space, the network learns to encode the images of
the person into pose-invariant latent codes, where the information is fused.
Second, based on the observation that feed-forward predictions are fast but do
not always align with the input images, we predict using both, bottom-up and
top-down streams (one per view) allowing information to flow in both
directions. Learning relies only on synthetic 3D data. Once learned, the model
can take a variable number of frames as input, and is able to reconstruct
shapes even from a single image with an accuracy of 6mm. Results on 3 different
datasets demonstrate the efficacy and accuracy of our approach.

本文提出了一种基于学习的模型，利用少量单目视频帧 (1-8 帧) 从移动的人中推断出其个性化的 3D 形状，可在不到 10 秒内达到 5mm 的重建精度。该模型学习预测一个统计体型模型的参数和实例化偏移量，同时加入衣服和头发的形状，具有快速和准确的预测，这得益于其对 T-pose 空间的预测和从上往下和从下往上两个视图预测的结合。该模型仅基于合成的 3D 数据进行学习，可将可变数量的帧作为输入，即使仅有一张图像，也能以 6mm 的精度重建形状。3 个不同数据集的结果表明了本方法的有效性和准确性。

从单个 RGB 相机学习重建穿着衣服的人

Learning to Reconstruct People in Clothing from a Single RGB Camera

Direct prediction of 3D body pose and shape remains a challenge even for
highly parameterized deep learning models. Mapping from the 2D image space to
the prediction space is difficult: perspective ambiguities make the loss
function noisy and training data is scarce. In this paper, we propose a novel
approach (Neural Body Fitting (NBF)). It integrates a statistical body model
within a CNN, leveraging reliable bottom-up semantic body part segmentation and
robust top-down body model constraints. NBF is fully differentiable and can be
trained using 2D and 3D annotations. In detailed experiments, we analyze how
the components of our model affect performance, especially the use of part
segmentations as an explicit intermediate representation, and present a robust,
efficiently trainable framework for 3D human pose estimation from 2D images
with competitive results on standard benchmarks. Code will be made available at
this http URL

该论文提出了一种新方法，将一个统计体模型嵌入到 CNN 中，并利用底部向上的语义体部分分割和稳健的自上而下的体模型约束，实现从 2D 图像中进行 3D 人体姿态估计的目的。

神经身体拟合：统一深度学习和基于模型的人体姿态和形状估计

Neural Body Fitting: Unifying Deep Learning and Model-Based Human Pose  and Shape Estimation

We address the problem of making human motion capture in the wild more
practical by using a small set of inertial sensors attached to the body. Since
the problem is heavily under-constrained, previous methods either use a large
number of sensors, which is intrusive, or they require additional video input.
We take a different approach and constrain the problem by: (i) making use of a
realistic statistical body model that includes anthropometric constraints and
(ii) using a joint optimization framework to fit the model to orientation and
acceleration measurements over multiple frames. The resulting tracker Sparse
Inertial Poser (SIP) enables 3D human pose estimation using only 6 sensors
(attached to the wrists, lower legs, back and head) and works for arbitrary
human motions. Experiments on the recently released TNT15 dataset show that,
using the same number of sensors, SIP achieves higher accuracy than the dataset
baseline without using any video data. We further demonstrate the effectiveness
of SIP on newly recorded challenging motions in outdoor scenarios such as
climbing or jumping over a wall.

本文提出了一种利用少量惯性传感器进行人体动作捕捉的方法，通过应用逼真的统计身体模型和使用关节优化框架来实现六个传感器进行任意人类动作的三维姿态估计，并在户外场景下的运动中展示了其高准确性。