End-to-end automatic speech recognition (E2E-ASR) can be classified into
several network architectures, such as connectionist temporal classification
(CTC), recurrent neural network transducer (RNN-T), attention-based
encoder-decoder, and mask-predict models. Each network architecture has
advantages and disadvantages, leading practitioners to switch between these
different models depending on application requirements. Instead of building
separate models, we propose a joint modeling scheme where four decoders (CTC,
RNN-T, attention, and mask-predict) share the same encoder -- we refer to this
as 4D modeling. The 4D model is trained using multitask learning, which will
bring model regularization and maximize the model robustness thanks to their
complementary properties. To efficiently train the 4D model, we introduce a
two-stage training strategy that stabilizes multitask learning. In addition, we
propose three novel one-pass beam search algorithms by combining three decoders
(CTC, RNN-T, and attention) to further improve performance. These three beam
search algorithms differ in which decoder is used as the primary decoder. We
carefully evaluate the performance and computational tradeoffs associated with
each algorithm. Experimental results demonstrate that the jointly trained 4D
model outperforms the E2E-ASR models trained with only one individual decoder.
Furthermore, we demonstrate that the proposed one-pass beam search algorithm
outperforms the previously proposed CTC/attention decoding.

我们提出了一种联合建模方案，通过多任务学习和三种新的单通道波束搜索算法，我们训练出的 4D 模型在端到端自动语音识别上取得了更好的性能，并超过了使用单个解码器的模型。