Text-to-video (T2V) synthesis has gained increasing attention in the
community, in which the recently emerged diffusion models (DMs) have
promisingly shown stronger performance than the past approaches. While existing
state-of-the-art DMs are competent to achieve high-resolution video generation,
they may largely suffer from key limitations (e.g., action occurrence
disorders, crude video motions) with respect to the intricate temporal dynamics
modeling, one of the crux of video synthesis. In this work, we investigate
strengthening the awareness of video dynamics for DMs, for high-quality T2V
generation. Inspired by human intuition, we design an innovative dynamic scene
manager (dubbed as Dysen) module, which includes (step-1) extracting from input
text the key actions with proper time-order arrangement, (step-2) transforming
the action schedules into the dynamic scene graph (DSG) representations, and
(step-3) enriching the scenes in the DSG with sufficient and reasonable
details. Taking advantage of the existing powerful LLMs (e.g., ChatGPT) via
in-context learning, Dysen realizes (nearly) human-level temporal dynamics
understanding. Finally, the resulting video DSG with rich action scene details
is encoded as fine-grained spatio-temporal features, integrated into the
backbone T2V DM for video generating. Experiments on popular T2V datasets
suggest that our framework consistently outperforms prior arts with significant
margins, especially in the scenario with complex actions. Project page at
this https URL

通过设计一个创新的动态场景管理器 (Dysen) 模块，该模块可以从输入文本中提取关键动作并将其以适当的时间顺序和动态场景图 (DSG) 表示转换，从而丰富了视频的场景细节，并与主干的 T2V DM 相结合来实现高质量的文本到视频生成。

利用大型语言模型增强面向动态感知的文本到视频扩散

Empowering Dynamics-aware Text-to-Video Diffusion with Large Language  Models

Although previous co-speech gesture generation methods are able to synthesize
motions in line with speech content, it is still not enough to handle diverse
and complicated motion distribution. The key challenges are: 1) the one-to-many
nature between the speech content and gestures; 2) the correlation modeling
between the body joints. In this paper, we present a novel framework (EMoG) to
tackle the above challenges with denoising diffusion models: 1) To alleviate
the one-to-many problem, we incorporate emotion clues to guide the generation
process, making the generation much easier; 2) To model joint correlation, we
propose to decompose the difficult gesture generation into two sub-problems:
joint correlation modeling and temporal dynamics modeling. Then, the two
sub-problems are explicitly tackled with our proposed Joint Correlation-aware
transFormer (JCFormer). Through extensive evaluations, we demonstrate that our
proposed method surpasses previous state-of-the-art approaches, offering
substantial superiority in gesture synthesis.

本文提出了一种通过加入情感线索指导生成过程、将难以生成的姿态生成分解为关节相关性建模和时间动力学建模两个子问题，然后分别使用提出的 JCFormer 模型来明确地解决子问题的新型框架 EMoG，该方法在手势合成方面表现优异，超过了以前的现有方法。