We present a new end-to-end architecture for automatic speech recognition
(ASR) that can be trained using \emph{symbolic} input in addition to the
traditional acoustic input. This architecture utilizes two separate encoders:
one for acoustic input and another for symbolic input, both sharing the
attention and decoder parameters. We call this architecture a multi-modal data
augmentation network (MMDA), as it can support multi-modal (acoustic and
symbolic) input and enables seamless mixing of large text datasets with
significantly smaller transcribed speech corpora during training. We study
different ways of transforming large text corpora into a symbolic form suitable
for training our MMDA network. Our best MMDA setup obtains small improvements
on character error rate (CER), and as much as 7-10\% relative word error rate
(WER) improvement over a baseline both with and without an external language
model.

本文提出了一种新的端到端自动语音识别（ASR）架构，可以利用符号输入和传统的声学输入进行训练，该架构使用两个单独的编码器：一个用于声学输入，另一个用于符号输入，并共享注意力和解码器参数；通过研究不同的方法将大型文本语料库转换成符号形式进行训练，我们的最佳 MMDA 设置不仅可以在字符错误率（CER）上获得小的改善，而且在基线上，无论是否有外部语言模型，均可以获得 7-10％相对词错误率（WER）的改进。