We propose a new transformer-based image and video tokenizer with Binary
Spherical Quantization (BSQ). BSQ projects the high-dimensional visual
embedding to a lower-dimensional hypersphere and then applies binary
quantization. BSQ is (1) parameter-efficient without an explicit codebook, (2)
scalable to arbitrary token dimensions, and (3) compact: compressing visual
data by up to 100$\times$ with minimal distortion. Our tokenizer uses a
transformer encoder and decoder with simple block-wise causal masking to
support variable-length videos as input. The resulting BSQ-ViT achieves
state-of-the-art visual reconstruction quality on image and video
reconstruction benchmarks with 2.4$\times$ throughput compared to the best
prior methods. Furthermore, by learning an autoregressive prior for adaptive
arithmetic coding, BSQ-ViT achieves comparable results on video compression
with state-of-the-art video compression standards. BSQ-ViT also enables masked
language models to achieve competitive image synthesis quality to GAN- and
diffusion-based methods.

提出了一种新的基于转换器的图像和视频分词器，使用二值球面量化实现。BSQ 将高维视觉嵌入投影到低维超球面上，然后应用二值量化。我们的分词器使用变长视频输入的转换器编码器和解码器，通过简单的分块因果掩蔽实现。基于此的 BSQ-ViT 在图像和视频重建基准上达到了最先进的视觉重建质量，并实现了最佳先前方法的 2.4 倍吞吐量。此外，通过学习自回归先验进行自适应算术编码，BSQ-ViT 在视频压缩方面达到了与最先进的视频压缩标准相当的结果。BSQ-ViT 还使得遮蔽语言模型能够实现与基于 GAN 和扩散的方法相媲美的图像合成质量。