Tokenizer, serving as a translator to map the intricate visual data into a
compact latent space, lies at the core of visual generative models. Based on
the finding that existing tokenizers are tailored to image or video inputs,
this paper presents OmniTokenizer, a transformer-based tokenizer for joint
image and video tokenization. OmniTokenizer is designed with a spatial-temporal
decoupled architecture, which integrates window and causal attention for
spatial and temporal modeling. To exploit the complementary nature of image and
video data, we further propose a progressive training strategy, where
OmniTokenizer is first trained on image data on a fixed resolution to develop
the spatial encoding capacity and then jointly trained on image and video data
on multiple resolutions to learn the temporal dynamics. OmniTokenizer, for the
first time, handles both image and video inputs within a unified framework and
proves the possibility of realizing their synergy. Extensive experiments
demonstrate that OmniTokenizer achieves state-of-the-art (SOTA) reconstruction
performance on various image and video datasets, e.g., 1.11 reconstruction FID
on ImageNet and 42 reconstruction FVD on UCF-101, beating the previous SOTA
methods by 13% and 26%, respectively. Additionally, we also show that when
integrated with OmniTokenizer, both language model-based approaches and
diffusion models can realize advanced visual synthesis performance,
underscoring the superiority and versatility of our method. Code is available
at this https URL

OmniTokenizer 是一种基于 Transformer 的 tokenizer，旨在处理图像和视频输入，并通过空间 - 时间解耦架构以及渐进式训练策略提供先进的重构性能。