Recent developments in large speech foundation models like Whisper have led
to their widespread use in many automatic speech recognition (ASR)
applications. These systems incorporate `special tokens' in their vocabulary,
such as $\texttt{<endoftext>}$, to guide their language generation process.
However, we demonstrate that these tokens can be exploited by adversarial
attacks to manipulate the model's behavior. We propose a simple yet effective
method to learn a universal acoustic realization of Whisper's
$\texttt{<endoftext>}$ token, which, when prepended to any speech signal,
encourages the model to ignore the speech and only transcribe the special
token, effectively `muting' the model. Our experiments demonstrate that the
same, universal 0.64-second adversarial audio segment can successfully mute a
target Whisper ASR model for over 97\% of speech samples. Moreover, we find
that this universal adversarial audio segment often transfers to new datasets
and tasks. Overall this work demonstrates the vulnerability of Whisper models
to `muting' adversarial attacks, where such attacks can pose both risks and
potential benefits in real-world settings: for example the attack can be used
to bypass speech moderation systems, or conversely the attack can also be used
to protect private speech data.

最近，诸如 Whisper 之类的大型语音基础模型的发展使其在许多自动语音识别（ASR）应用中广泛使用。然而，我们证明了这些特殊令牌可以被对抗攻击利用以操纵模型的行为。我们提出了一种简单但有效的方法来学习 Whisper 的通用声学实现的 <endoftext> 令牌，当添加到任何语音信号之前时，鼓励模型忽略语音并仅转录特殊令牌，有效地 “静音” 模型。我们的实验证明，同一个通用的 0.64 秒对抗音频片段可以成功地使目标 Whisper ASR 模型在 97％以上的语音样本中静音。此外，我们发现这个通用的对抗音频片段通常可迁移到新的数据集和任务上。总的来说，这项工作证明了 Whisper 模型对 “静音” 对抗性攻击的脆弱性，其中这些攻击在现实世界的环境中既可能有风险，也可能有潜在的益处：例如，攻击可以用来绕过语音调节系统，或者反之，攻击也可以用来保护私人语音数据。