In end-to-end (E2E) speech recognition models, a representational
tight-coupling inevitably emerges between the encoder and the decoder. We build
upon recent work that has begun to explore building encoders with modular
encoded representations, such that encoders and decoders from different models
can be stitched together in a zero-shot manner without further fine-tuning.
While previous research only addresses full-context speech models, we explore
the problem in a streaming setting as well. Our framework builds on top of
existing encoded representations, converting them to modular features, dubbed
as Lego-Features, without modifying the pre-trained model. The features remain
interchangeable when the model is retrained with distinct initializations.
Though sparse, we show that the Lego-Features are powerful when tested with
RNN-T or LAS decoders, maintaining high-quality downstream performance. They
are also rich enough to represent the first-pass prediction during two-pass
deliberation. In this scenario, they outperform the N-best hypotheses, since
they do not need to be supplemented with acoustic features to deliver the best
results. Moreover, generating the Lego-Features does not require beam search or
auto-regressive computation. Overall, they present a modular, powerful and
cheap alternative to the standard encoder output, as well as the N-best
hypotheses.

该研究探讨了使用模块化编码表示建立模型的编码器以及使用 Lego-Features 作为可重用的模块化特征以提高语音识别性能，同时不需要使用自动回归计算或波束搜索。