As a robust and large-scale multilingual speech recognition model, Whisper
has demonstrated impressive results in many low-resource and
out-of-distribution scenarios. However, its encoder-decoder structure hinders
its application to streaming speech recognition. In this paper, we introduce
Simul-Whisper, which uses the time alignment embedded in Whisper's
cross-attention to guide auto-regressive decoding and achieve chunk-based
streaming ASR without any fine-tuning of the pre-trained model. Furthermore, we
observe the negative effect of the truncated words at the chunk boundaries on
the decoding results and propose an integrate-and-fire-based truncation
detection model to address this issue. Experiments on multiple languages and
Whisper architectures show that Simul-Whisper achieves an average absolute word
error rate degradation of only 1.46% at a chunk size of 1 second, which
significantly outperforms the current state-of-the-art baseline.

Simul-Whisper is a streaming speech recognition model that utilizes time alignment embedded in Whisper's cross-attention for guiding auto-regressive decoding, achieving chunk-based ASR without fine-tuning, while proposing an integrate-and-fire-based truncation detection model to address the negative effect of truncated words at chunk boundaries, outperforming the current state-of-the-art baseline with a minimal absolute word error rate degradation.