We propose to replace vector quantization (VQ) in the latent representation
of VQ-VAEs with a simple scheme termed finite scalar quantization (FSQ), where
we project the VAE representation down to a few dimensions (typically less than
10). Each dimension is quantized to a small set of fixed values, leading to an
(implicit) codebook given by the product of these sets. By appropriately
choosing the number of dimensions and values each dimension can take, we obtain
the same codebook size as in VQ. On top of such discrete representations, we
can train the same models that have been trained on VQ-VAE representations. For
example, autoregressive and masked transformer models for image generation,
multimodal generation, and dense prediction computer vision tasks. Concretely,
we employ FSQ with MaskGIT for image generation, and with UViM for depth
estimation, colorization, and panoptic segmentation. Despite the much simpler
design of FSQ, we obtain competitive performance in all these tasks. We
emphasize that FSQ does not suffer from codebook collapse and does not need the
complex machinery employed in VQ (commitment losses, codebook reseeding, code
splitting, entropy penalties, etc.) to learn expressive discrete
representations.

我们提出使用有限数量标量量化（FSQ）替代 VQ-VAEs 中的向量量化（VQ），通过将 VAE 表示投影到少量维度中，并将每个维度量化为固定值的小集合，从而实现离散表示。我们在图像生成中使用 MaskGIT，以及在深度估计、上色和全景分割中使用 UViM，尽管 FSQ 的设计简单，但在这些任务中获得竞争性表现。我们强调 FSQ 不会出现代码本崩溃问题，也不需要复杂的机制来学习表达丰富的离散表示。

有限数量的标量量化：简化 VQ-VAE

Finite Scalar Quantization: VQ-VAE Made Simple

Far beyond learning long-range interactions of natural language, transformers
are becoming the de-facto standard for many vision tasks with their power and
scalability. Especially with cross-modal tasks between image and text, vector
quantized variational autoencoders (VQ-VAEs) are widely used to make a raw RGB
image into a sequence of feature vectors. To better leverage the correlation
between image and text, we propose L-Verse, a novel architecture consisting of
feature-augmented variational autoencoder (AugVAE) and bidirectional
auto-regressive transformer (BiART) for image-to-text and text-to-image
generation. Our AugVAE shows the state-of-the-art reconstruction performance on
ImageNet1K validation set, along with the robustness to unseen images in the
wild. Unlike other models, BiART can distinguish between image (or text) as a
conditional reference and a generation target. L-Verse can be directly used for
image-to-text or text-to-image generation without any finetuning or extra
object detection framework. In quantitative and qualitative experiments,
L-Verse shows impressive results against previous methods in both image-to-text
and text-to-image generation on MS-COCO Captions. We furthermore assess the
scalability of L-Verse architecture on Conceptual Captions and present the
initial result of bidirectional vision-language representation learning on
general domain.

L-Verse 是一种图像文本双向生成神经网络，它由增强特征自编码器 AugVAE 和双向自回归变压器 BiART 组成，能够在不需要调整或额外目标检测框架的情况下直接用于图像到文本或文本到图像的生成。

L-Verse：图像与文本之间的双向生成

L-Verse: Bidirectional Generation Between Image and Text

Despite progress in training neural networks for lossy image compression,
current approaches fail to maintain both perceptual quality and abstract
features at very low bitrates. Encouraged by recent success in learning
discrete representations with Vector Quantized Variational Autoencoders
(VQ-VAEs), we motivate the use of a hierarchy of VQ-VAEs to attain high factors
of compression. We show that the combination of stochastic quantization and
hierarchical latent structure aids likelihood-based image compression. This
leads us to introduce a novel objective for training hierarchical VQ-VAEs. Our
resulting scheme produces a Markovian series of latent variables that
reconstruct images of high-perceptual quality which retain semantically
meaningful features. We provide qualitative and quantitative evaluations on the
CelebA and MNIST datasets.

该研究使用层次 VQ-VAE 近似压缩图像，同时保持感知质量和抽象特征。在 CelebA 和 MNIST 数据集上进行了定性和定量评估。