Whisper is a multitask and multilingual speech model covering 99 languages.
It yields commendable automatic speech recognition (ASR) results in a subset of
its covered languages, but the model still underperforms on a non-negligible
number of under-represented languages, a problem exacerbated in smaller model
versions. In this work, we examine its limitations, demonstrating the presence
of speaker-related (gender, age) and model-related (resourcefulness and model
size) bias. Despite that, we show that only model-related bias are amplified by
quantization, impacting more low-resource languages and smaller models.
Searching for a better compression approach, we propose DistilWhisper, an
approach that is able to bridge the performance gap in ASR for these languages
while retaining the advantages of multitask and multilingual capabilities. Our
approach involves two key strategies: lightweight modular ASR fine-tuning of
whisper-small using language-specific experts, and knowledge distillation from
whisper-large-v2. This dual approach allows us to effectively boost ASR
performance while keeping the robustness inherited from the multitask and
multilingual pre-training. Results demonstrate that our approach is more
effective than standard fine-tuning or LoRA adapters, boosting performance in
the targeted languages for both in- and out-of-domain test sets, while
introducing only a negligible parameter overhead at inference.

在这项研究中，我们发现了 Whisper 这个多任务和多语言语音模型在少数语言上的性能问题，并证明这是与讲话者特征及模型相关偏差有关。针对这个问题，我们提出了 DistilWhisper 方法，通过轻量级的模型微调和知识蒸馏策略，在保留了多任务和多语言预训练的鲁棒性的同时，有效地提升了目标语言的 ASR 性能。

多任务多语言语音模型的高效压缩

Efficient Compression of Multitask Multilingual Speech Models

Whisper is a multitask and multilingual speech model covering 99 languages.
It yields commendable automatic speech recognition (ASR) results in a subset of
its covered languages, but the model still under-performs on a non-negligible
number of under-represented languages, a problem exacerbated in smaller model
versions. In this work, we propose DistilWhisper, an approach able to bridge
the performance gap in ASR for these languages while retaining the advantages
of multitask and multilingual capabilities. Our approach involves two key
strategies: lightweight modular ASR fine-tuning of whisper-small using
language-specific experts, and knowledge distillation from whisper-large-v2.
This dual approach allows us to effectively boost ASR performance while keeping
the robustness inherited from the multitask and multilingual pre-training.
Results demonstrate that our approach is more effective than standard
fine-tuning or LoRA adapters, boosting performance in the targeted languages
for both in- and out-of-domain test sets, while introducing only a negligible
parameter overhead at inference.

DistilWhisper 是一个轻量级模块化的 ASR 框架，通过语言专家和知识蒸馏的方式提高了 Whisper 的性能，同时保留了多任务和多语言预训练模型的鲁棒性，在目标语言中提高了 ASR 性能而仅引入了可忽略的参数开销。