Existing vector quantization (VQ) based autoregressive models follow a
two-stage generation paradigm that first learns a codebook to encode images as
discrete codes, and then completes generation based on the learned codebook.
However, they encode fixed-size image regions into fixed-length codes and
ignore their naturally different information densities, which results in
insufficiency in important regions and redundancy in unimportant ones, and
finally degrades the generation quality and speed. Moreover, the fixed-length
coding leads to an unnatural raster-scan autoregressive generation. To address
the problem, we propose a novel two-stage framework: (1) Dynamic-Quantization
VAE (DQ-VAE) which encodes image regions into variable-length codes based on
their information densities for an accurate and compact code representation.
(2) DQ-Transformer which thereby generates images autoregressively from
coarse-grained (smooth regions with fewer codes) to fine-grained (details
regions with more codes) by modeling the position and content of codes in each
granularity alternately, through a novel stacked-transformer architecture and
shared-content, non-shared position input layers designs. Comprehensive
experiments on various generation tasks validate our superiorities in both
effectiveness and efficiency. Code will be released at
this https URL

提出了一种新的两阶段框架，它使用动态量化 VAE 将图像区域编码为基于其信息密度的可变长度代码，以实现更准确，更紧凑的代码表示，从而解决了现有 VQ 基于自回归模型的固定长度编码问题，该方法通过一种新的堆叠 Transformer 架构和共享内容、独立位置输入层设计，从粗粒度到细粒度生成图像。