This work evaluated several cutting-edge large-scale foundation models based
on self-supervision or weak supervision, including SeamlessM4T, SeamlessM4T v2,
and Whisper-large-v3, on three code-switched corpora. We found that
self-supervised models can achieve performances close to the supervised model,
indicating the effectiveness of multilingual self-supervised pre-training. We
also observed that these models still have room for improvement as they kept
making similar mistakes and had unsatisfactory performances on modeling
intra-sentential code-switching. In addition, the validity of several variants
of Whisper was explored, and we concluded that they remained effective in a
code-switching scenario, and similar techniques for self-supervised models are
worth studying to boost the performance of code-switched tasks.

本文评估了几种基于自监督或弱监督的尖端大型基础模型（包括 SeamlessM4T、SeamlessM4T v2 和 Whisper-large-v3）在三个混合代码语料库上的表现。我们发现自监督模型可以达到接近受监督模型的性能，表明多语言自监督预训练的有效性。我们还观察到这些模型在建模句内代码切换方面仍有改进空间，常犯相似错误并在代码切换任务的性能上表现不理想。此外，我们探索了 Whisper 的几种变体的有效性，并得出结论它们在代码切换场景中仍然有效，鼓励研究类似的自监督模型技术以提升代码切换任务的性能。

探索基于自监督和弱监督的最新基础模型在汉英混合语音识别和语音转文字翻译的零迁移能力

Investigating Zero-Shot Generalizability on Mandarin-English  Code-Switched ASR and Speech-to-text Translation of Recent Foundation Models  with Self-Supervision and Weak Supervision