Vector quantization (VQ) is a key technique in high-resolution and
high-fidelity image synthesis, which aims to learn a codebook to encode an
image with a sequence of discrete codes and then generate an image in an
auto-regression manner. Although existing methods have shown superior
performance, most methods prefer to learn a single-modal codebook (\emph{e.g.},
image), resulting in suboptimal performance when the codebook is applied to
multi-modal downstream tasks (\emph{e.g.}, text-to-image, image captioning) due
to the existence of modal gaps. In this paper, we propose a novel
language-guided codebook learning framework, called LG-VQ, which aims to learn
a codebook that can be aligned with the text to improve the performance of
multi-modal downstream tasks. Specifically, we first introduce pre-trained text
semantics as prior knowledge, then design two novel alignment modules
(\emph{i.e.}, Semantic Alignment Module, and Relationship Alignment Module) to
transfer such prior knowledge into codes for achieving codebook text alignment.
In particular, our LG-VQ method is model-agnostic, which can be easily
integrated into existing VQ models. Experimental results show that our method
achieves superior performance on reconstruction and various multi-modal
downstream tasks.

本文提出了一种新颖的语言引导的码书学习框架 (LG-VQ)，旨在学习与文本对齐的码书，以提高多模态下游任务的性能。实验结果表明，我们的方法在重构和各种多模态下游任务上取得了优越的性能。