In NLP, text language models based on words or subwords are known to
outperform their character-based counterparts. Yet, in the speech community,
the standard input of spoken LMs are 20ms or 40ms-long discrete units (shorter
than a phoneme). Taking inspiration from word-based LM, we introduce a
Generative Spoken Language Model (GSLM) based on word-size continuous-valued
audio embeddings that can generate diverse and expressive language output. This
is obtained by replacing lookup table for lexical types with a Lexical
Embedding function, the cross entropy loss by a contrastive loss, and
multinomial sampling by k-NN sampling. The resulting model is the first
generative language model based on word-size continuous embeddings. Its
performance is on par with discrete unit GSLMs regarding generation quality as
measured by automatic metrics and subjective human judgements. Moreover, it is
five times more memory efficient thanks to its large 200ms units. In addition,
the embeddings before and after the Lexical Embedder are phonetically and
semantically interpretable.

基于连续值音频嵌入的生成式口语语言模型（GSLM）通过引入词大小连续嵌入函数、对比损失和 k-NN 采样，取得了多样性和富有表现力的语言生成；该模型与离散单元 GSLM 在生成质量方面表现相当，同时内存效率提高了五倍；此外，词嵌入前后的嵌入具有音韵和语义解释性。

基于连续词大小的音频令牌的生成式口语模型

Generative Spoken Language Model based on continuous word-sized audio  tokens

Speech representations learned from Self-supervised learning (SSL) models can
benefit various speech processing tasks. However, utilizing SSL representations
usually requires fine-tuning the pre-trained models or designing task-specific
downstream models and loss functions, causing much memory usage and human
labor. Recently, prompting in Natural Language Processing (NLP) has been found
to be an efficient technique to leverage pre-trained language models (LMs).
Specifically, prompt tuning optimizes a limited number of task-specific
parameters with a fixed pre-trained model; as a result, only a small set of
parameters is needed to be stored for each task. Prompt tuning improves
computation and memory efficiency by leveraging the pre-trained LM's prediction
ability. Nevertheless, such a paradigm is little studied in the speech
community. We report in this paper the first exploration of the prompt tuning
paradigm for speech processing tasks based on Generative Spoken Language Model
(GSLM). Experiment results show that the prompt tuning technique achieves
competitive performance in speech classification tasks with fewer trainable
parameters than fine-tuning specialized downstream models. We further study the
technique in challenging sequence generation tasks. Prompt tuning also
demonstrates its potential, while the limitation and possible research
directions are discussed in this paper. The source code is available on
this https URL.

本研究首次探讨了基于生成式口语语言模型 (GSLM) 的提示调整范式用于语音处理任务，实验结果表明，与下游精细调整模型相比，提示调整技术使用的可调参数更少，在语音分类任务中实现了较高性能。