Text and vision foundation models can perform many tasks in a zero-shot
setting, a desirable property that enables these systems to be applied in
general and low-resource settings. However, there has been significantly less
work on the zero-shot abilities of ASR foundation models, with these systems
typically fine-tuned to specific tasks or constrained to applications that
match their training criterion and data annotation. In this work we investigate
the ability of Whisper and MMS, ASR foundation models trained primarily for
speech recognition, to perform zero-shot audio classification. We use simple
template-based text prompts at the decoder and use the resulting decoding
probabilities to generate zero-shot predictions. Without training the model on
extra data or adding any new parameters, we demonstrate that Whisper shows
promising zero-shot classification performance on a range of 8
audio-classification datasets, outperforming existing state-of-the-art
zero-shot baseline's accuracy by an average of 9%. One important step to unlock
the emergent ability is debiasing, where a simple unsupervised reweighting
method of the class probabilities yields consistent significant performance
gains. We further show that performance increases with model size, implying
that as ASR foundation models scale up, they may exhibit improved zero-shot
performance.

使用 Whisper 这个 ASR 基础模型进行无监督音频分类，并通过去偏方法获得显著的性能提升。