In this paper, we propose an efficient and accurate streaming speech
recognition model based on the FastConformer architecture. We adapted the
FastConformer architecture for streaming applications through: (1) constraining
both the look-ahead and past contexts in the encoder, and (2) introducing an
activation caching mechanism to enable the non-autoregressive encoder to
operate autoregressively during inference. The proposed model is thoughtfully
designed in a way to eliminate the accuracy disparity between the train and
inference time which is common for many streaming models. Furthermore, our
proposed encoder works with various decoder configurations including
Connectionist Temporal Classification (CTC) and RNN-Transducer (RNNT) decoders.
Additionally, we introduced a hybrid CTC/RNNT architecture which utilizes a
shared encoder with both a CTC and RNNT decoder to boost the accuracy and save
computation. We evaluate the proposed model on LibriSpeech dataset and a
multi-domain large scale dataset and demonstrate that it can achieve better
accuracy with lower latency and inference time compared to a conventional
buffered streaming model baseline. We also showed that training a model with
multiple latencies can achieve better accuracy than single latency models while
it enables us to support multiple latencies with a single model. Our
experiments also showed the hybrid architecture would not only speedup the
convergence of the CTC decoder but also improves the accuracy of streaming
models compared to single decoder models.

我们提出了一种基于 FastConformer 架构的高效准确的流式语音识别模型，在编码器中通过约束前向和后向上下文，并引入了激活缓存机制来实现非自回归编码器在推断过程中的自回归操作。我们的模型消除了许多流式模型中常见的训练与推断准确性差异，并能与多种解码器配置一起工作。此外，我们还引入了一种混合 CTC/RNNT 架构，利用共享编码器与 CTC 和 RNNT 解码器结合，以提高准确性并节省计算资源。我们在 LibriSpeech 数据集和多领域大规模数据集上评估了我们的模型，并证明它相对于传统的缓冲流式模型基线具有更高的准确性、更低的延迟和推断时间。同时，我们的实验证明使用多种延迟训练模型可以获得比单一延迟模型更高的准确性，并能够通过一个模型支持多种延迟。我们的实验还表明，相比于单一解码器模型，混合架构不仅加快了 CTC 解码器的收敛速度，而且提高了流式模型的准确性。