With recent rapid growth of large language models (LLMs), discrete speech
tokenization has played an important role for injecting speech into LLMs.
However, this discretization gives rise to a loss of information, consequently
impairing overall performance. To improve the performance of these discrete
speech tokens, we present RepCodec, a novel speech representation codec for
semantic speech tokenization. In contrast to audio codecs which reconstruct the
raw audio, RepCodec learns a vector quantization codebook through
reconstructing speech representations from speech encoders like HuBERT or
data2vec. Together, the speech encoder, the codec encoder and the vector
quantization codebook form a pipeline for converting speech waveforms into
semantic tokens. The extensive experiments illustrate that RepCodec, by virtue
of its enhanced information retention capacity, significantly outperforms the
widely used k-means clustering approach in both speech understanding and
generation. Furthermore, this superiority extends across various speech
encoders and languages, affirming the robustness of RepCodec. We believe our
method can facilitate large language modeling research on speech processing.

通过学习语音编码器、编解码器和向量量化码本，RepCodec 通过语义语音标记化将语音波形转换为语义标记，从而显著提升语音理解和生成的性能。