Text-conditioned diffusion models have emerged as a promising tool for neural
video generation. However, current models still struggle with intricate
spatiotemporal prompts and often generate restricted or incorrect motion (e.g.,
even lacking the ability to be prompted for objects moving from left to right).
To address these limitations, we introduce LLM-grounded Video Diffusion (LVD).
Instead of directly generating videos from the text inputs, LVD first leverages
a large language model (LLM) to generate dynamic scene layouts based on the
text inputs and subsequently uses the generated layouts to guide a diffusion
model for video generation. We show that LLMs are able to understand complex
spatiotemporal dynamics from text alone and generate layouts that align closely
with both the prompts and the object motion patterns typically observed in the
real world. We then propose to guide video diffusion models with these layouts
by adjusting the attention maps. Our approach is training-free and can be
integrated into any video diffusion model that admits classifier guidance. Our
results demonstrate that LVD significantly outperforms its base video diffusion
model and several strong baseline methods in faithfully generating videos with
the desired attributes and motion patterns.

在视频生成方面，本研究提出了一种基于文本的扩散模型，通过使用大型语言模型（LLM）生成的动态场景布局来引导视频生成过程，旨在解决现有模型在处理复杂时空提示时遇到的限制和错误运动的问题。实验证明该方法在生成具有所需特征和运动模式的视频方面显著优于基准模型和一些强基线方法。