Recently Whisper has approached human-level robustness and accuracy in English automatic speech recognition (ASR), while in minor language and mixed language speech recognition, there remains a compelling need for further improvement. In this work, we present the impressive results of Whisper-MCE, our finetuned Whisper model, which was trained using our self-collected dataset, Mixed Cantonese and English audio dataset (MCE). Meanwhile, considering word error rate (WER) poses challenges when it comes to evaluating its effectiveness in minor language and mixed-language contexts, we present a novel rating mechanism. By comparing our model to the baseline whisper-large-v2 model, we demonstrate its superior ability to accurately capture the content of the original audio, achieve higher recognition accuracy, and exhibit faster recognition speed. Notably, our model outperforms other existing models in the specific task of recognizing mixed language.

本文介绍了Whisper-MCE模型在minor language和mixed language语音识别方面的卓越表现，同时提出一种新的评估机制以解决在这些语境下对其有效性进行评估时的挑战。通过将我们的模型与基线模型whisper-large-v2进行比较，我们展示了其准确捕捉原始音频内容、实现更高的识别准确率和更快的识别速度的卓越能力。值得注意的是，在识别mixed language的特定任务中，我们的模型超越了其他现有模型。

Whisper-MCE：用于混合语言更佳性能的Whisper模型微调