We present ART$\boldsymbol{\cdot}$V, an efficient framework for
auto-regressive video generation with diffusion models. Unlike existing methods
that generate entire videos in one-shot, ART$\boldsymbol{\cdot}$V generates a
single frame at a time, conditioned on the previous ones. The framework offers
three distinct advantages. First, it only learns simple continual motions
between adjacent frames, therefore avoiding modeling complex long-range motions
that require huge training data. Second, it preserves the high-fidelity
generation ability of the pre-trained image diffusion models by making only
minimal network modifications. Third, it can generate arbitrarily long videos
conditioned on a variety of prompts such as text, image or their combinations,
making it highly versatile and flexible. To combat the common drifting issue in
AR models, we propose masked diffusion model which implicitly learns which
information can be drawn from reference images rather than network predictions,
in order to reduce the risk of generating inconsistent appearances that cause
drifting. Moreover, we further enhance generation coherence by conditioning it
on the initial frame, which typically contains minimal noise. This is
particularly useful for long video generation. When trained for only two weeks
on four GPUs, ART$\boldsymbol{\cdot}$V already can generate videos with natural
motions, rich details and a high level of aesthetic quality. Besides, it
enables various appealing applications, e.g., composing a long video from
multiple text prompts.

ART・V 是一个高效的自回归视频生成框架，使用扩散模型逐帧生成视频，并通过简化连续帧之间的运动来避免建模复杂的远程运动，同时保留预训练图像扩散模型的高保真生成能力。通过结合文本、图像等多种提示，ART・V 可以生成任意长度的视频，具有高度的多功能性和灵活性。通过引入掩蔽扩散模型和使用初始帧作为条件，ART・V 进一步提升了生成一致性和生成的视觉质量，能够在短时间内生成带有自然运动、丰富细节和高美感的视频，并支持多个文本提示组合成长视频的应用。