In this paper, we present a novel two-pass approach to unify streaming and
non-streaming end-to-end (E2E) speech recognition in a single model. Our model
adopts the hybrid CTC/attention architecture, in which the conformer layers in
the encoder are modified. We propose a dynamic chunk-based attention strategy
to allow arbitrary right context length. At inference time, the CTC decoder
generates n-best hypotheses in a streaming way. The inference latency could be
easily controlled by only changing the chunk size. The CTC hypotheses are then
rescored by the attention decoder to get the final result. This efficient
rescoring process causes very little sentence-level latency. Our experiments on
the open 170-hour AISHELL-1 dataset show that, the proposed method can unify
the streaming and non-streaming model simply and efficiently. On the AISHELL-1
test set, our unified model achieves 5.60% relative character error rate (CER)
reduction in non-streaming ASR compared to a standard non-streaming
transformer. The same model achieves 5.42% CER with 640ms latency in a
streaming ASR system.

本文提出了一种新颖的两步方法，用于将流式和非流式端到端（E2E）语音识别统一到单个模型中，该模型采用混合 CTC /attention 架构，在编码器的 conformer 层进行修改，并且在推理时，CTC 解码器以流式方式生成 n 个最佳假设，然后通过注意力解码器进行重评分以得到最终结果，并且在 AISHELL-1 测试集上，基于提出的方法的统一模型相对于标准的非流式 transformer，实现了 5.60％的相对字符错误率（CER）降低，在流式 ASR 系统中实现了 640ms 的延迟。