We present Semantify: a self-supervised method that utilizes the semantic
power of CLIP language-vision foundation model to simplify the control of 3D
morphable models. Given a parametric model, training data is created by
randomly sampling the model's parameters, creating various shapes and rendering
them. The similarity between the output images and a set of word descriptors is
calculated in CLIP's latent space. Our key idea is first to choose a small set
of semantically meaningful and disentangled descriptors that characterize the
3DMM, and then learn a non-linear mapping from scores across this set to the
parametric coefficients of the given 3DMM. The non-linear mapping is defined by
training a neural network without a human-in-the-loop. We present results on
numerous 3DMMs: body shape models, face shape and expression models, as well as
animal shapes. We demonstrate how our method defines a simple slider interface
for intuitive modeling, and show how the mapping can be used to instantly fit a
3D parametric body shape to in-the-wild images.

Semantify 是一种自监督方法，利用 CLIP 语言 - 视觉基础模型的语义能力来简化对 3D 可塑模型的控制。通过随机抽样模型参数创建训练数据，并计算输出图像与一组词汇描述符之间在 CLIP 潜空间中的相似性，我们通过首先选择一小组语义有意义且脱离的描述符来表征 3DMM，然后学习一个非线性映射从这组描述符的分数到给定 3DMM 的参数系数。这个非线性映射通过训练一个无需人工干预的神经网络来定义。我们展示了在多个 3DMM 上的结果，包括身体形状模型、面部形状和表情模型以及动物形状。演示了我们的方法如何定义一个简单的滑块界面以实现直观建模，并展示了该映射如何将 3D 参数化身体形状立即适配到实际图像。