Vector Quantized Variational AutoEncoder (VQ-VAE) is an established technique
in machine learning for learning discrete representations across various
modalities. However, its scalability and applicability are limited by the need
to retrain the model to adjust the codebook for different data or model scales.
We introduce the Rate-Adaptive VQ-VAE (RAQ-VAE) framework, which addresses this
challenge with two novel codebook representation methods: a model-based
approach using a clustering-based technique on an existing well-trained VQ-VAE
model, and a data-driven approach utilizing a sequence-to-sequence (Seq2Seq)
model for variable-rate codebook generation. Our experiments demonstrate that
RAQ-VAE achieves effective reconstruction performance across multiple rates,
often outperforming conventional fixed-rate VQ-VAE models. This work enhances
the adaptability and performance of VQ-VAEs, with broad applications in data
reconstruction, generation, and computer vision tasks.

Rate-Adaptive VQ-VAE improves the adaptability and performance of Vector Quantized Variational AutoEncoders with novel codebook representation methods, achieving effective reconstruction performance across multiple rates.