Conformer models maintain a large number of internal states, the vast
majority of which are associated with self-attention layers. With limited
memory bandwidth, reading these from memory at each inference step can slow
down inference. In this paper, we design an optimized conformer that is small
enough to meet on-device restrictions and has fast inference on TPUs. We
explore various ideas to improve the execution speed, including replacing lower
conformer blocks with convolution-only blocks, strategically downsizing the
architecture, and utilizing an RNNAttention-Performer. Our optimized conformer
can be readily incorporated into a cascaded-encoder setting, allowing a
second-pass decoder to operate on its output and improve the accuracy whenever
more resources are available. Altogether, we find that these optimizations can
reduce latency by a factor of 6.8x, and come at a reasonable trade-off in
quality. With the cascaded second-pass, we show that the recognition accuracy
is completely recoverable. Thus, our proposed encoder can double as a strong
standalone encoder in on device, and as the first part of a high-performance
ASR pipeline.

本文提出了一种优化的 conformer 模型，通过替换低层 conformer 块、策略性缩小架构和利用 RNNAttention-Performer 等方式，优化内部状态数量，降低推理延迟。通过级联编码器，我们发现这些优化可以将延迟降低 6.8 倍，并保持合适的精度，可以作为独立的编码器或高性能 ASR 流水线的第一部分。