We present SCULPT, a novel 3D generative model for clothed and textured 3D
meshes of humans. Specifically, we devise a deep neural network that learns to
represent the geometry and appearance distribution of clothed human bodies.
Training such a model is challenging, as datasets of textured 3D meshes for
humans are limited in size and accessibility. Our key observation is that there
exist medium-sized 3D scan datasets like CAPE, as well as large-scale 2D image
datasets of clothed humans and multiple appearances can be mapped to a single
geometry. To effectively learn from the two data modalities, we propose an
unpaired learning procedure for pose-dependent clothed and textured human
meshes. Specifically, we learn a pose-dependent geometry space from 3D scan
data. We represent this as per vertex displacements w.r.t. the SMPL model.
Next, we train a geometry conditioned texture generator in an unsupervised way
using the 2D image data. We use intermediate activations of the learned
geometry model to condition our texture generator. To alleviate entanglement
between pose and clothing type, and pose and clothing appearance, we condition
both the texture and geometry generators with attribute labels such as clothing
types for the geometry, and clothing colors for the texture generator. We
automatically generated these conditioning labels for the 2D images based on
the visual question answering model BLIP and CLIP. We validate our method on
the SCULPT dataset, and compare to state-of-the-art 3D generative models for
clothed human bodies. We will release the codebase for research purposes.

我们提出了 SCULPT，一种新颖的针对穿着衣服和贴图的人体 3D 网格的生成模型。