This article describes an efficient end-to-end speech translation (E2E-ST)
framework based on non-autoregressive (NAR) models. End-to-end speech
translation models have several advantages over traditional cascade systems
such as inference latency reduction. However, conventional AR decoding methods
are not fast enough because each token is generated incrementally. NAR models,
however, can accelerate the decoding speed by generating multiple tokens in
parallel on the basis of the token-wise conditional independence assumption. We
propose a unified NAR E2E-ST framework called Orthros, which has an NAR decoder
and an auxiliary shallow AR decoder on top of the shared encoder. The auxiliary
shallow AR decoder selects the best hypothesis by rescoring multiple candidates
generated from the NAR decoder in parallel (parallel AR rescoring). We adopt
conditional masked language model (CMLM) and a connectionist temporal
classification (CTC)-based model as NAR decoders for Orthros, referred to as
Orthros-CMLM and Orthros-CTC, respectively. We also propose two training
methods to enhance the CMLM decoder. Experimental evaluations on three
benchmark datasets with six language directions demonstrated that Orthros
achieved large improvements in translation quality with a very small overhead
compared with the baseline NAR model. Moreover, the Conformer encoder
architecture enabled large quality improvements, especially for CTC-based
models. Orthros-CTC with the Conformer encoder increased decoding speed by
3.63x on CPU with translation quality comparable to that of an AR model.

本文提出了一种基于非自回归模型 (NAR) 的高效端到端语音翻译 (E2E-ST) 系统 ——Orthros，使用条件掩蔽语言模型 (CMLM) 和连接主义时间分类 (CTC) 模型作为 NAR 解码器，并采用两种训练方法来增强 CMLM 解码器。实验表明，在三个基准数据集及六个语言方向上，Orthros-CTC 以 Conformer 编码器为基础的模型在保证翻译质量的前提下，将解码速度提高了 3.63 倍。