Motions in a video primarily consist of camera motion, induced by camera
movement, and object motion, resulting from object movement. Accurate control
of both camera and object motion is essential for video generation. However,
existing works either mainly focus on one type of motion or do not clearly
distinguish between the two, limiting their control capabilities and diversity.
Therefore, this paper presents MotionCtrl, a unified and flexible motion
controller for video generation designed to effectively and independently
control camera and object motion. The architecture and training strategy of
MotionCtrl are carefully devised, taking into account the inherent properties
of camera motion, object motion, and imperfect training data. Compared to
previous methods, MotionCtrl offers three main advantages: 1) It effectively
and independently controls camera motion and object motion, enabling more
fine-grained motion control and facilitating flexible and diverse combinations
of both types of motion. 2) Its motion conditions are determined by camera
poses and trajectories, which are appearance-free and minimally impact the
appearance or shape of objects in generated videos. 3) It is a relatively
generalizable model that can adapt to a wide array of camera poses and
trajectories once trained. Extensive qualitative and quantitative experiments
have been conducted to demonstrate the superiority of MotionCtrl over existing
methods.

该论文介绍了 MotionCtrl，这是一个用于视频生成的统一和灵活的运动控制器，通过有效和独立地控制相机运动和物体运动，实现更精细的运动控制和促进两种类型运动的灵活多样组合。