In recent years, diffusion models have made remarkable strides in
text-to-video generation, sparking a quest for enhanced control over video
outputs to more accurately reflect user intentions. Traditional efforts
predominantly focus on employing either semantic cues, like images or depth
maps, or motion-based conditions, like moving sketches or object bounding
boxes. Semantic inputs offer a rich scene context but lack detailed motion
specificity; conversely, motion inputs provide precise trajectory information
but miss the broader semantic narrative. For the first time, we integrate both
semantic and motion cues within a diffusion model for video generation, as
demonstrated in Fig 1. To this end, we introduce the Scene and Motion
Conditional Diffusion (SMCD), a novel methodology for managing multimodal
inputs. It incorporates a recognized motion conditioning module and
investigates various approaches to integrate scene conditions, promoting
synergy between different modalities. For model training, we separate the
conditions for the two modalities, introducing a two-stage training pipeline.
Experimental results demonstrate that our design significantly enhances video
quality, motion precision, and semantic coherence.

通过将语义和动作线索整合到扩散模型中，我们引入了一种用于视频生成的新方法，它显著提升了视频质量、动作精度和语义连贯性。