We introduce MotionCLIP, a 3D human motion auto-encoder featuring a latent
embedding that is disentangled, well behaved, and supports highly semantic
textual descriptions. MotionCLIP gains its unique power by aligning its latent
space with that of the Contrastive Language-Image Pre-training (CLIP) model.
Aligning the human motion manifold to CLIP space implicitly infuses the
extremely rich semantic knowledge of CLIP into the manifold. In particular, it
helps continuity by placing semantically similar motions close to one another,
and disentanglement, which is inherited from the CLIP-space structure.
MotionCLIP comprises a transformer-based motion auto-encoder, trained to
reconstruct motion while being aligned to its text label's position in
CLIP-space. We further leverage CLIP's unique visual understanding and inject
an even stronger signal through aligning motion to rendered frames in a
self-supervised manner. We show that although CLIP has never seen the motion
domain, MotionCLIP offers unprecedented text-to-motion abilities, allowing
out-of-domain actions, disentangled editing, and abstract language
specification. For example, the text prompt "couch" is decoded into a sitting
down motion, due to lingual similarity, and the prompt "Spiderman" results in a
web-swinging-like solution that is far from seen during training. In addition,
we show how the introduced latent space can be leveraged for motion
interpolation, editing and recognition.

MotionCLIP 是一种 3D 人体运动自编码器，在 latent space 中对接 CLIP 模型，以获得无与伦比的文本到运动的能力，实现了跨域动作，编辑和抽象语言规范等功能。