Spoken language understanding systems using audio-only data are gaining
popularity, yet their ability to handle unseen intents remains limited. In this
study, we propose a generalized zero-shot audio-to-intent classification
framework with only a few sample text sentences per intent. To achieve this, we
first train a supervised audio-to-intent classifier by making use of a
self-supervised pre-trained model. We then leverage a neural audio synthesizer
to create audio embeddings for sample text utterances and perform generalized
zero-shot classification on unseen intents using cosine similarity. We also
propose a multimodal training strategy that incorporates lexical information
into the audio representation to improve zero-shot performance. Our multimodal
training approach improves the accuracy of zero-shot intent classification on
unseen intents of SLURP by 2.75% and 18.2% for the SLURP and internal
goal-oriented dialog datasets, respectively, compared to audio-only training.

利用仅有每个意图几个样本文本句子的泛化零样本音频到意图分类框架，该框架使用只有音频的数据，通过训练有监督的音频到意图分类器和利用神经音频合成器生成音频嵌入，使用余弦相似度对未见过的意图进行泛化零样本分类，并通过多模态训练策略将词汇信息融入音频表征来提高泛化零样本性能。相较于仅使用音频训练，我们的多模态训练方法提高了 SLURP 和内部目标导向对话数据集上未见过的意图的零样本意图分类准确率分别为 2.75％和 18.2％。

泛零样本音频到意图分类

Generalized zero-shot audio-to-intent classification

Recent progress in deep learning for audio synthesis opens the way to models
that directly produce the waveform, shifting away from the traditional paradigm
of relying on vocoders or MIDI synthesizers for speech or music generation.
Despite their successes, current state-of-the-art neural audio synthesizers
such as WaveNet and SampleRNN suffer from prohibitive training and inference
times because they are based on autoregressive models that generate audio
samples one at a time at a rate of 16kHz. In this work, we study the more
computationally efficient alternative of generating the waveform frame-by-frame
with large strides. We present SING, a lightweight neural audio synthesizer for
the original task of generating musical notes given desired instrument, pitch
and velocity. Our model is trained end-to-end to generate notes from nearly
1000 instruments with a single decoder, thanks to a new loss function that
minimizes the distances between the log spectrograms of the generated and
target waveforms. On the generalization task of synthesizing notes for pairs of
pitch and instrument not seen during training, SING produces audio with
significantly improved perceptual quality compared to a state-of-the-art
autoencoder based on WaveNet as measured by a Mean Opinion Score (MOS), and is
about 32 times faster for training and 2, 500 times faster for inference.

本研究提出了一种基于深度学习的轻量级神经音频合成器 SING，其可以通过单一解码器从近 1000 种乐器中生成音符，并且通过新的损失函数将生成的和目标波形的对数谱距离最小化，提高了音质的感知品质。与基于 WaveNet 的最新自动编码器相比，在训练和推理方面分别快 32 倍和快 2500 倍。