In this paper, we define and study a new Cloth2Body problem which has a goal
of generating 3D human body meshes from a 2D clothing image. Unlike the
existing human mesh recovery problem, Cloth2Body needs to address new and
emerging challenges raised by the partial observation of the input and the high
diversity of the output. Indeed, there are three specific challenges. First,
how to locate and pose human bodies into the clothes. Second, how to
effectively estimate body shapes out of various clothing types. Finally, how to
generate diverse and plausible results from a 2D clothing image. To this end,
we propose an end-to-end framework that can accurately estimate 3D body mesh
parameterized by pose and shape from a 2D clothing image. Along this line, we
first utilize Kinematics-aware Pose Estimation to estimate body pose
parameters. 3D skeleton is employed as a proxy followed by an inverse
kinematics module to boost the estimation accuracy. We additionally design an
adaptive depth trick to align the re-projected 3D mesh better with 2D clothing
image by disentangling the effects of object size and camera extrinsic. Next,
we propose Physics-informed Shape Estimation to estimate body shape parameters.
3D shape parameters are predicted based on partial body measurements estimated
from RGB image, which not only improves pixel-wise human-cloth alignment, but
also enables flexible user editing. Finally, we design Evolution-based pose
generation method, a skeleton transplanting method inspired by genetic
algorithms to generate diverse reasonable poses during inference. As shown by
experimental results on both synthetic and real-world data, the proposed
framework achieves state-of-the-art performance and can effectively recover
natural and diverse 3D body meshes from 2D images that align well with
clothing.

在本研究中，我们定义和研究了一个新的 Cloth2Body 问题，其目标是从 2D 服装图像生成 3D 人体网格。与现有的人体网格恢复问题不同，Cloth2Body 需要解决输入的局部观察和输出的高度多样性带来的新挑战。我们提出了一个端到端的框架，从 2D 服装图像准确估计参数化为姿势和形状的 3D 体网格，并通过实验结果证明该框架达到了最先进的性能，在对齐服装的同时能够有效恢复自然而多样化的 3D 人体网格。