Spoken language understanding systems using audio-only data are gaining
popularity, yet their ability to handle unseen intents remains limited. In this
study, we propose a generalized zero-shot audio-to-intent classification
framework with only a few sample text sentences per intent. To achieve this, we
first train a supervised audio-to-intent classifier by making use of a
self-supervised pre-trained model. We then leverage a neural audio synthesizer
to create audio embeddings for sample text utterances and perform generalized
zero-shot classification on unseen intents using cosine similarity. We also
propose a multimodal training strategy that incorporates lexical information
into the audio representation to improve zero-shot performance. Our multimodal
training approach improves the accuracy of zero-shot intent classification on
unseen intents of SLURP by 2.75% and 18.2% for the SLURP and internal
goal-oriented dialog datasets, respectively, compared to audio-only training.

利用仅有每个意图几个样本文本句子的泛化零样本音频到意图分类框架，该框架使用只有音频的数据，通过训练有监督的音频到意图分类器和利用神经音频合成器生成音频嵌入，使用余弦相似度对未见过的意图进行泛化零样本分类，并通过多模态训练策略将词汇信息融入音频表征来提高泛化零样本性能。相较于仅使用音频训练，我们的多模态训练方法提高了 SLURP 和内部目标导向对话数据集上未见过的意图的零样本意图分类准确率分别为 2.75％和 18.2％。