Videos of actions are complex signals containing rich compositional structure
in space and time. Current video generation methods lack the ability to
condition the generation on multiple coordinated and potentially simultaneous
timed actions. To address this challenge, we propose to represent the actions
in a graph structure called Action Graph and present the new ``Action Graph To
Video'' synthesis task. Our generative model for this task (AG2Vid)
disentangles motion and appearance features, and by incorporating a scheduling
mechanism for actions facilitates a timely and coordinated video generation. We
train and evaluate AG2Vid on the CATER and Something-Something V2 datasets, and
show that the resulting videos have better visual quality and semantic
consistency compared to baselines. Finally, our model demonstrates zero-shot
abilities by synthesizing novel compositions of the learned actions. For code
and pretrained models, see the project page this https URL

本文首次提出 Action Graph 结构来实现多协调及可能同时进行的时间动作相关视频的生成方式，通过 AG2Vid 模型实现了运动和外观特征的分离，并引入了调度机制以协助视频生成，经过对 CATER 和 Something-Something V2 数据集的训练和评估，证明与基线相比，所生成的视频具有更好的视觉质量和语义一致性。同时，该模型还展现了零样本能力，在生成已学习动作的新组合等方面表现出良好的效果。