We present VideoGPT: a conceptually simple architecture for scaling
likelihood based generative modeling to natural videos. VideoGPT uses VQ-VAE
that learns downsampled discrete latent representations of a raw video by
employing 3D convolutions and axial self-attention. A simple GPT-like
architecture is then used to autoregressively model the discrete latents using
spatio-temporal position encodings. Despite the simplicity in formulation and
ease of training, our architecture is able to generate samples competitive with
state-of-the-art GAN models for video generation on the BAIR Robot dataset, and
generate high fidelity natural videos from UCF-101 and Tumbler GIF Dataset
(TGIF). We hope our proposed architecture serves as a reproducible reference
for a minimalistic implementation of transformer based video generation models.
Samples and code are available at
this https URL

VideoGPT 是一种简单的架构，利用 VQ-VAE 学习离散化的原始视频的下采样离散潜在表示，并使用类似于 GPT 的简单架构来自回归地模拟其离散潜在分布，以帮助生成高保真的自然视频。