End-to-end (E2E) automatic speech recognition (ASR) can operate in two modes:
streaming and non-streaming, each with its pros and cons. Streaming ASR
processes the speech frames in real-time as it is being received, while
non-streaming ASR waits for the entire speech utterance; thus, professionals
may have to operate in either mode to satisfy their application. In this work,
we present joint optimization of streaming and non-streaming ASR based on
multi-decoder and knowledge distillation. Primarily, we study 1) the encoder
integration of these ASR modules, followed by 2) separate decoders to make the
switching mode flexible, and enhancing performance by 3) incorporating
similarity-preserving knowledge distillation between the two modular encoders
and decoders. Evaluation results show 2.6%-5.3% relative character error rate
reductions (CERR) on CSJ for streaming ASR, and 8.3%-9.7% relative CERRs for
non-streaming ASR within a single model compared to multiple standalone
modules.

本文通过使用多解码器和知识蒸馏来实现流式和非流式自动语音识别（ASR）的联合优化，主要研究了 ASR 模块的编码器整合、分离解码器以实现灵活切换模式，以及通过两个模块化编码器和解码器之间的相似性保持知识蒸馏来提高性能。评估结果显示，在单一模型中与多个独立模块相比，流式 ASR 的字符错误率相对减少了 2.6%-5.3%，非流式 ASR 的字符错误率相对减少了 8.3%-9.7%。

多解码器与知识蒸馏的流式与非流式自动语音识别的联合优化

Joint Optimization of Streaming and Non-Streaming Automatic Speech  Recognition with Multi-Decoder and Knowledge Distillation

In this paper, we present a novel two-pass approach to unify streaming and
non-streaming end-to-end (E2E) speech recognition in a single model. Our model
adopts the hybrid CTC/attention architecture, in which the conformer layers in
the encoder are modified. We propose a dynamic chunk-based attention strategy
to allow arbitrary right context length. At inference time, the CTC decoder
generates n-best hypotheses in a streaming way. The inference latency could be
easily controlled by only changing the chunk size. The CTC hypotheses are then
rescored by the attention decoder to get the final result. This efficient
rescoring process causes very little sentence-level latency. Our experiments on
the open 170-hour AISHELL-1 dataset show that, the proposed method can unify
the streaming and non-streaming model simply and efficiently. On the AISHELL-1
test set, our unified model achieves 5.60% relative character error rate (CER)
reduction in non-streaming ASR compared to a standard non-streaming
transformer. The same model achieves 5.42% CER with 640ms latency in a
streaming ASR system.

本文提出了一种新颖的两步方法，用于将流式和非流式端到端（E2E）语音识别统一到单个模型中，该模型采用混合 CTC /attention 架构，在编码器的 conformer 层进行修改，并且在推理时，CTC 解码器以流式方式生成 n 个最佳假设，然后通过注意力解码器进行重评分以得到最终结果，并且在 AISHELL-1 测试集上，基于提出的方法的统一模型相对于标准的非流式 transformer，实现了 5.60％的相对字符错误率（CER）降低，在流式 ASR 系统中实现了 640ms 的延迟。