For autoregressive (AR) modeling of high-resolution images, vector
quantization (VQ) represents an image as a sequence of discrete codes. A short
sequence length is important for an AR model to reduce its computational costs
to consider long-range interactions of codes. However, we postulate that
previous VQ cannot shorten the code sequence and generate high-fidelity images
together in terms of the rate-distortion trade-off. In this study, we propose
the two-stage framework, which consists of Residual-Quantized VAE (RQ-VAE) and
RQ-Transformer, to effectively generate high-resolution images. Given a fixed
codebook size, RQ-VAE can precisely approximate a feature map of an image and
represent the image as a stacked map of discrete codes. Then, RQ-Transformer
learns to predict the quantized feature vector at the next position by
predicting the next stack of codes. Thanks to the precise approximation of
RQ-VAE, we can represent a 256$\times$256 image as 8$\times$8 resolution of the
feature map, and RQ-Transformer can efficiently reduce the computational costs.
Consequently, our framework outperforms the existing AR models on various
benchmarks of unconditional and conditional image generation. Our approach also
has a significantly faster sampling speed than previous AR models to generate
high-quality images.

本文提出了一种两阶段的框架，由残差量化 VAE（RQ-VAE）和 RQ-Transformer 组成，旨在有效地生成高分辨率图像，通过精确的逼近，我们可以将 256×256 像素的图像表示为 8×8 特征映射，并且 RQ-Transformer 可以在保证生成高品质图像的同时减少计算成本。