Large-scale auto-regressive language models pretrained on massive text have
demonstrated their impressive ability to perform new natural language tasks
with only a few text examples, without the need for fine-tuning. Recent studies
further show that such a few-shot learning ability can be extended to the
text-image setting by training an encoder to encode the images into embeddings
functioning like the text embeddings of the language model. Interested in
exploring the possibility of transferring the few-shot learning ability to the
audio-text setting, we propose a novel speech understanding framework,
WavPrompt, where we finetune a wav2vec model to generate a sequence of audio
embeddings understood by the language model. We show that WavPrompt is a
few-shot learner that can perform speech understanding tasks better than a
naive text baseline. We conduct detailed ablation studies on different
components and hyperparameters to empirically identify the best model
configuration. In addition, we conduct a non-speech understanding experiment to
show WavPrompt can extract more information than just the transcriptions. Code
is available at this https URL

介绍了一种名为 WavPrompt 的语音理解框架，该框架使用预先训练的自回归语言模型，通过微调一种 wav2vec 模型生成一系列音频嵌入来实现在音频文本场景下的 few-shot 学习能力。实验表明，WavPrompt 在执行语音理解任务时表现出比朴素的文本基线更好的表现，并且能够提取更多的信息。