End-to-end (E2E) automatic speech recognition (ASR) can operate in two modes:
streaming and non-streaming, each with its pros and cons. Streaming ASR
processes the speech frames in real-time as it is being received, while
non-streaming ASR waits for the entire speech utterance; thus, professionals
may have to operate in either mode to satisfy their application. In this work,
we present joint optimization of streaming and non-streaming ASR based on
multi-decoder and knowledge distillation. Primarily, we study 1) the encoder
integration of these ASR modules, followed by 2) separate decoders to make the
switching mode flexible, and enhancing performance by 3) incorporating
similarity-preserving knowledge distillation between the two modular encoders
and decoders. Evaluation results show 2.6%-5.3% relative character error rate
reductions (CERR) on CSJ for streaming ASR, and 8.3%-9.7% relative CERRs for
non-streaming ASR within a single model compared to multiple standalone
modules.

本文通过使用多解码器和知识蒸馏来实现流式和非流式自动语音识别（ASR）的联合优化，主要研究了 ASR 模块的编码器整合、分离解码器以实现灵活切换模式，以及通过两个模块化编码器和解码器之间的相似性保持知识蒸馏来提高性能。评估结果显示，在单一模型中与多个独立模块相比，流式 ASR 的字符错误率相对减少了 2.6%-5.3%，非流式 ASR 的字符错误率相对减少了 8.3%-9.7%。