This research explores the interaction between Whisper, a high-performing
speech recognition model, and information in prompts. Our results unexpectedly
show that Whisper may not fully grasp textual prompts as anticipated.
Additionally, we find that performance improvement is not guaranteed even with
stronger adherence to the topic information in textual prompts. It is also
noted that English prompts generally outperform Mandarin ones on datasets of
both languages, likely due to differences in training data distributions for
these languages. Conversely, we discover that Whisper exhibits awareness of
misleading information in language tokens by effectively ignoring incorrect
language tokens and focusing on the correct ones. In summary, this work raises
questions about Whisper's prompt understanding capability and encourages
further studies.

本研究探讨了高性能语音识别模型 Whisper 与提示信息之间的交互作用，结果意外发现 Whisper 可能并未完全理解文字提示。此外，我们发现即使更加严格地遵循文字提示中的主题信息，也不能保证性能提升。英文提示通常在两种语言的数据集上表现优于中文提示，这很可能是由于这些语言的训练数据分布存在差异所致。与此相反，我们发现 Whisper 能够意识到语言令牌中误导性信息，通过有效地忽略错误的语言令牌并专注于正确的令牌。总结起来，本研究提出了有关 Whisper 提示理解能力的问题，并鼓励进一步研究。