Prompting and adapter tuning have emerged as efficient alternatives to
fine-tuning (FT) methods. However, existing studies on speech prompting focused
on classification tasks and failed on more complex sequence generation tasks.
Besides, adapter tuning is primarily applied with a focus on encoder-only
self-supervised models. Our experiments show that prompting on Wav2Seq, a
self-supervised encoder-decoder model, surpasses previous works in sequence
generation tasks. It achieves a remarkable 53% relative improvement in word
error rate for ASR and a 27% in F1 score for slot filling. Additionally,
prompting competes with the FT method in the low-resource scenario. Moreover,
we show the transferability of prompting and adapter tuning on Wav2Seq in
cross-lingual ASR. When limited trainable parameters are involved, prompting
and adapter tuning consistently outperform conventional FT across 7 languages.
Notably, in the low-resource scenario, prompting consistently outperforms
adapter tuning.

通过在 Wav2Seq 模型上进行提示和适配器调优，我们在序列生成任务中取得了显著的成果，尤其在 ASR 的词错误率和槽位填充的 F1 分数上相对于往前的工作分别提高了 53% 和 27%。在 7 种语言中，当可训练参数有限时，提示和适配器调优表现优于传统的微调方法，尤其在资源匮乏的情况下，提示的表现更好。

用于自监督编码器 - 解码器语音模型的提示和适配器调整

Prompting and Adapter Tuning for Self-supervised Encoder-Decoder Speech  Model

We introduce Wav2Seq, the first self-supervised approach to pre-train both
parts of encoder-decoder models for speech data. We induce a pseudo language as
a compact discrete representation, and formulate a self-supervised pseudo
speech recognition task -- transcribing audio inputs into pseudo subword
sequences. This process stands on its own, or can be applied as low-cost
second-stage pre-training. We experiment with automatic speech recognition
(ASR), spoken named entity recognition, and speech-to-text translation. We set
new state-of-the-art results for end-to-end spoken named entity recognition,
and show consistent improvements on 20 language pairs for speech-to-text
translation, even when competing methods use additional text data for training.
Finally, on ASR, our approach enables encoder-decoder methods to benefit from
pre-training for all parts of the network, and shows comparable performance to
highly optimized recent methods.

Wav2Seq 是第一个用于预训练语音数据的自监督方法，采用了伪语言作为紧凑的离散表示，并制定了自监督伪语音识别任务 - 将音频输入转录为伪子词序列。