Recent advancements in generative models have highlighted the crucial role of
image tokenization in the efficient synthesis of high-resolution images.
Tokenization, which transforms images into latent representations, reduces
computational demands compared to directly processing pixels and enhances the
effectiveness and efficiency of the generation process. Prior methods, such as
VQGAN, typically utilize 2D latent grids with fixed downsampling factors.
However, these 2D tokenizations face challenges in managing the inherent
redundancies present in images, where adjacent regions frequently display
similarities. To overcome this issue, we introduce Transformer-based
1-Dimensional Tokenizer (TiTok), an innovative approach that tokenizes images
into 1D latent sequences. TiTok provides a more compact latent representation,
yielding substantially more efficient and effective representations than
conventional techniques. For example, a 256 x 256 x 3 image can be reduced to
just 32 discrete tokens, a significant reduction from the 256 or 1024 tokens
obtained by prior methods. Despite its compact nature, TiTok achieves
competitive performance to state-of-the-art approaches. Specifically, using the
same generator framework, TiTok attains 1.97 gFID, outperforming MaskGIT
baseline significantly by 4.21 at ImageNet 256 x 256 benchmark. The advantages
of TiTok become even more significant when it comes to higher resolution. At
ImageNet 512 x 512 benchmark, TiTok not only outperforms state-of-the-art
diffusion model DiT-XL/2 (gFID 2.74 vs. 3.04), but also reduces the image
tokens by 64x, leading to 410x faster generation process. Our best-performing
variant can significantly surpasses DiT-XL/2 (gFID 2.13 vs. 3.04) while still
generating high-quality samples 74x faster.

这篇研究论文介绍了一种基于 Transformer 的一维令牌化方法（TiTok），其将图像令牌化为一维潜在序列，通过提供更紧凑的潜在表示形式，实现了比传统技术更高效和更有效的图像合成。

一幅图像对于重建与生成而言价值 32 个令牌

An Image is Worth 32 Tokens for Reconstruction and Generation

Vision Transformer models process input images by dividing them into a
spatially regular grid of equal-size patches. Conversely, Transformers were
originally introduced over natural language sequences, where each token
represents a subword - a chunk of raw data of arbitrary size. In this work, we
apply this approach to Vision Transformers by introducing a novel image
tokenization scheme, replacing the standard uniform grid with a
mixed-resolution sequence of tokens, where each token represents a patch of
arbitrary size. Using the Quadtree algorithm and a novel saliency scorer, we
construct a patch mosaic where low-saliency areas of the image are processed in
low resolution, routing more of the model's capacity to important image
regions. Using the same architecture as vanilla ViTs, our Quadformer models
achieve substantial accuracy gains on image classification when controlling for
the computational budget. Code and models are publicly available at
this https URL .

本文介绍了一种新的图像标记方案 —— 混合分辨率标记 —— 以及如何使用 Quadtree 算法和新的显著性评分器构建图像的代表通量，进而实现更好的图像分类。