Generating controllable videos conforming to user intentions is an appealing
yet challenging topic in computer vision. To enable maneuverable control in
line with user intentions, a novel video generation task, named
Text-Image-to-Video generation (TI2V), is proposed. With both controllable
appearance and motion, TI2V aims at generating videos from a static image and a
text description. The key challenges of TI2V task lie both in aligning
appearance and motion from different modalities, and in handling uncertainty in
text descriptions. To address these challenges, we propose a Motion
Anchor-based video GEnerator (MAGE) with an innovative motion anchor (MA)
structure to store appearance-motion aligned representation. To model the
uncertainty and increase the diversity, it further allows the injection of
explicit condition and implicit randomness. Through three-dimensional axial
transformers, MA is interacted with given image to generate next frames
recursively with satisfying controllability and diversity. Accompanying the new
task, we build two new video-text paired datasets based on MNIST and CATER for
evaluation. Experiments conducted on these datasets verify the effectiveness of
MAGE and show appealing potentials of TI2V task. Source code for model and
datasets will be available soon.

提出了一种名为 TI2V 的新的视频生成任务，即从静态图像和文本描述生成视频。介绍了一种名为 MAGE 的视频生成器，其中包含一种创新的运动锚（MA）结构，以存储外观 - 运动对齐表示，通过三维轴向变压器与给定图像交互，支持可控性和多样性。两个新的视频 - 文本匹配数据集验证了 MAGE 的有效性并展示了 TI2V 的潜力。