The essence of a video lies in its dynamic motions, including character
actions, object movements, and camera movements. While text-to-video generative
diffusion models have recently advanced in creating diverse contents,
controlling specific motions through text prompts remains a significant
challenge. A primary issue is the coupling of appearance and motion, often
leading to overfitting on appearance. To tackle this challenge, we introduce
MotionCrafter, a novel one-shot instance-guided motion customization method.
MotionCrafter employs a parallel spatial-temporal architecture that injects the
reference motion into the temporal component of the base model, while the
spatial module is independently adjusted for character or style control. To
enhance the disentanglement of motion and appearance, we propose an innovative
dual-branch motion disentanglement approach, comprising a motion
disentanglement loss and an appearance prior enhancement strategy. During
training, a frozen base model provides appearance normalization, effectively
separating appearance from motion and thereby preserving diversity.
Comprehensive quantitative and qualitative experiments, along with user
preference tests, demonstrate that MotionCrafter can successfully integrate
dynamic motions while preserving the coherence and quality of the base model
with a wide range of appearance generation capabilities. Codes are available at
this https URL

MotionCrafter 是一种新型一次性实例引导的动作定制方法，通过并行的时空架构注入参考动作到基础模型的时间部分，同时独立调整空间模块以进行角色或风格控制，以增强动作与外观的解耦，进而维持多样性，量化和定性实验证明了 MotionCrafter 在整合动态动作、保持一致性和提供广泛外观生成能力方面的成功。

MotionCrafter：一次性定制扩散模型的运动

MotionCrafter: One-Shot Motion Customization of Diffusion Models

In this paper, we address unsupervised pose-guided person image generation,
which is known challenging due to non-rigid deformation. Unlike previous
methods learning a rock-hard direct mapping between human bodies, we propose a
new pathway to decompose the hard mapping into two more accessible subtasks,
namely, semantic parsing transformation and appearance generation. Firstly, a
semantic generative network is proposed to transform between semantic parsing
maps, in order to simplify the non-rigid deformation learning. Secondly, an
appearance generative network learns to synthesize semantic-aware textures.
Thirdly, we demonstrate that training our framework in an end-to-end manner
further refines the semantic maps and final results accordingly. Our method is
generalizable to other semantic-aware person image generation tasks, eg,
clothing texture transfer and controlled image manipulation. Experimental
results demonstrate the superiority of our method on DeepFashion and
Market-1501 datasets, especially in keeping the clothing attributes and better
body shapes.

本文提出了一种无监督学习的姿态引导的人物图像生成方法，通过将生成人体的硬映射分解为语义解析和外貌生成两个子任务的形式，实现了人体的变形生成。