Recent text-to-video diffusion models have achieved impressive progress. In
practice, users often desire the ability to control object motion and camera
movement independently for customized video creation. However, current methods
lack the focus on separately controlling object motion and camera movement in a
decoupled manner, which limits the controllability and flexibility of
text-to-video models. In this paper, we introduce Direct-a-Video, a system that
allows users to independently specify motions for one or multiple objects
and/or camera movements, as if directing a video. We propose a simple yet
effective strategy for the decoupled control of object motion and camera
movement. Object motion is controlled through spatial cross-attention
modulation using the model's inherent priors, requiring no additional
optimization. For camera movement, we introduce new temporal cross-attention
layers to interpret quantitative camera movement parameters. We further employ
an augmentation-based approach to train these layers in a self-supervised
manner on a small-scale dataset, eliminating the need for explicit motion
annotation. Both components operate independently, allowing individual or
combined control, and can generalize to open-domain scenarios. Extensive
experiments demonstrate the superiority and effectiveness of our method.
Project page: this https URL

我们引入 Direct-a-Video，一个可以让用户独立指定一个或多个对象的运动和 / 或相机的移动，就像导演一个视频一样。通过使用模型的先验知识，我们提出了一种简单又有效的对象运动和相机移动解耦控制策略。我们还引入了新的时间交叉注意力层来解释定量的相机移动参数。通过在小规模数据集上自监督方式训练这些层，我们消除了显式运动注释的需求。我们的方法能够独立运作，并能够广泛适用于开放领域的场景。大量实验证明了我们方法的优越性和有效性。