Text-to-video (T2V) generation is a rapidly growing research area that aims
to translate the scenes, objects, and actions within complex video text into a
sequence of coherent visual frames. We present FlowZero, a novel framework that
combines Large Language Models (LLMs) with image diffusion models to generate
temporally-coherent videos. FlowZero uses LLMs to understand complex
spatio-temporal dynamics from text, where LLMs can generate a comprehensive
dynamic scene syntax (DSS) containing scene descriptions, object layouts, and
background motion patterns. These elements in DSS are then used to guide the
image diffusion model for video generation with smooth object motions and
frame-to-frame coherence. Moreover, FlowZero incorporates an iterative
self-refinement process, enhancing the alignment between the spatio-temporal
layouts and the textual prompts for the videos. To enhance global coherence, we
propose enriching the initial noise of each frame with motion dynamics to
control the background movement and camera motion adaptively. By using
spatio-temporal syntaxes to guide the diffusion process, FlowZero achieves
improvement in zero-shot video synthesis, generating coherent videos with vivid
motion.

Text-to-video generation using FlowZero, a framework that combines Large Language Models (LLMs) with image diffusion models, achieves improvement in zero-shot video synthesis by generating coherent videos with vivid motion.