Vector-Quantized Image Modeling (VQIM) is a fundamental research problem in
image synthesis, which aims to represent an image with a discrete token
sequence. Existing studies effectively address this problem by learning a
discrete codebook from scratch and in a code-independent manner to quantize
continuous representations into discrete tokens. However, learning a codebook
from scratch and in a code-independent manner is highly challenging, which may
be a key reason causing codebook collapse, i.e., some code vectors can rarely
be optimized without regard to the relationship between codes and good codebook
priors such that die off finally. In this paper, inspired by pretrained
language models, we find that these language models have actually pretrained a
superior codebook via a large number of text corpus, but such information is
rarely exploited in VQIM. To this end, we propose a novel codebook transfer
framework with part-of-speech, called VQCT, which aims to transfer a
well-trained codebook from pretrained language models to VQIM for robust
codebook learning. Specifically, we first introduce a pretrained codebook from
language models and part-of-speech knowledge as priors. Then, we construct a
vision-related codebook with these priors for achieving codebook transfer.
Finally, a novel codebook transfer network is designed to exploit abundant
semantic relationships between codes contained in pretrained codebooks for
robust VQIM codebook learning. Experimental results on four datasets show that
our VQCT method achieves superior VQIM performance over previous
state-of-the-art methods.

通过预训练语言模型中的先前训练的编码簿以及词性知识作为先验，构建一个视觉相关的编码簿，通过新的编码簿传递网络来实现鲁棒的 VQIM 编码簿学习，实验证明我们的 VQCT 方法在四个数据集上取得了优于现有最先进方法的性能。

利用词性的码书迁移在矢量量化图像建模中的应用

Codebook Transfer with Part-of-Speech for Vector-Quantized Image  Modeling

Vector-quantized image modeling has shown great potential in synthesizing
high-quality images. However, generating high-resolution images remains a
challenging task due to the quadratic computational overhead of the
self-attention process. In this study, we seek to explore a more efficient
two-stage framework for high-resolution image generation with improvements in
the following three aspects. (1) Based on the observation that the first
quantization stage has solid local property, we employ a local attention-based
quantization model instead of the global attention mechanism used in previous
methods, leading to better efficiency and reconstruction quality. (2) We
emphasize the importance of multi-grained feature interaction during image
generation and introduce an efficient attention mechanism that combines global
attention (long-range semantic consistency within the whole image) and local
attention (fined-grained details). This approach results in faster generation
speed, higher generation fidelity, and improved resolution. (3) We propose a
new generation pipeline incorporating autoencoding training and autoregressive
generation strategy, demonstrating a better paradigm for image synthesis.
Extensive experiments demonstrate the superiority of our approach in
high-quality and high-resolution image reconstruction and generation.

通过引入局部关注机制、融合全局关注和局部关注机制、以及自动编码训练和自回归生成策略，提出了一种高分辨率图像生成的高效两阶段框架，实现了更高效率、更好的重建质量和更高分辨率的图像合成。