Vector Quantisation (VQ) is experiencing a comeback in machine learning,
where it is increasingly used in representation learning. However, optimizing
the codevectors in existing VQ-VAE is not entirely trivial. A problem is
codebook collapse, where only a small subset of codevectors receive gradients
useful for their optimisation, whereas a majority of them simply ``dies off''
and is never updated or used. This limits the effectiveness of VQ for learning
larger codebooks in complex computer vision tasks that require high-capacity
representations. In this paper, we present a simple alternative method for
online codebook learning, Clustering VQ-VAE (CVQ-VAE). Our approach selects
encoded features as anchors to update the ``dead'' codevectors, while
optimising the codebooks which are alive via the original loss. This strategy
brings unused codevectors closer in distribution to the encoded features,
increasing the likelihood of being chosen and optimized. We extensively
validate the generalization capability of our quantiser on various datasets,
tasks (e.g. reconstruction and generation), and architectures (e.g. VQ-VAE,
VQGAN, LDM). Our CVQ-VAE can be easily integrated into the existing models with
just a few lines of code.

参考文献通过在线聚类学习方法 Clustering VQ-VAE（CVQ-VAE）解决了向量量化（VQ）中的代码书坍塌问题，提高了在复杂的计算机视觉任务中学习更大的代码书的效果。

在线聚类码本

Online Clustered Codebook

Background: Computational models of speech recognition often assume that the
set of target words is already given. This implies that these models do not
learn to recognise speech from scratch without prior knowledge and explicit
supervision. Visually grounded speech models learn to recognise speech without
prior knowledge by exploiting statistical dependencies between spoken and
visual input. While it has previously been shown that visually grounded speech
models learn to recognise the presence of words in the input, we explicitly
investigate such a model as a model of human speech recognition.
Methods: We investigate the time-course of word recognition as simulated by
the model using a gating paradigm to test whether its recognition is affected
by well-known word-competition effects in human speech processing. We
furthermore investigate whether vector quantisation, a technique for discrete
representation learning, aids the model in the discovery and recognition of
words.
Results/Conclusion: Our experiments show that the model is able to recognise
nouns in isolation and even learns to properly differentiate between plural and
singular nouns. We also find that recognition is influenced by word competition
from the word-initial cohort and neighbourhood density, mirroring word
competition effects in human speech comprehension. Lastly, we find no evidence
that vector quantisation is helpful in discovering and recognising words. Our
gating experiments even show that the vector quantised model requires more of
the input sequence for correct recognition.

该研究使用视觉以及语音的统计依赖关系进行词语识别，并探究其中的单复数形式和词汇竞争效应，同时研究了矢量量化在识别词语方面是否有益。