Motion-based controllable text-to-video generation involves motions to
control the video generation. Previous methods typically require the training
of models to encode motion cues or the fine-tuning of video diffusion models.
However, these approaches often result in suboptimal motion generation when
applied outside the trained domain. In this work, we propose MotionClone, a
training-free framework that enables motion cloning from a reference video to
control text-to-video generation. We employ temporal attention in video
inversion to represent the motions in the reference video and introduce primary
temporal-attention guidance to mitigate the influence of noisy or very subtle
motions within the attention weights. Furthermore, to assist the generation
model in synthesizing reasonable spatial relationships and enhance its
prompt-following capability, we propose a location-aware semantic guidance
mechanism that leverages the coarse location of the foreground from the
reference video and original classifier-free guidance features to guide the
video generation. Extensive experiments demonstrate that MotionClone exhibits
proficiency in both global camera motion and local object motion, with notable
superiority in terms of motion fidelity, textual alignment, and temporal
consistency.

MotionClone 是一个无需训练的框架，通过时间注意力机制从参考视频中克隆运动以控制文本到视频生成，同时引入了区域感知的语义引导机制提升生成模型的空间关系和随机应变能力，实验结果显示 MotionClone 在全局摄像机运动和局部目标运动方面表现出色，并具有运动保真度高、文本对齐准确和时序一致等显著优势。