Attention-based sequence-to-sequence modeling provides a powerful and elegant
solution for applications that need to map one sequence to a different
sequence. Its success heavily relies on the availability of large amounts of
training data. This presents a challenge for speech applications where labelled
speech data is very expensive to obtain, such as automatic speech recognition
(ASR) and speech translation (ST). In this study, we propose a general
multi-task learning framework to leverage text data for ASR and ST tasks. Two
auxiliary tasks, a denoising autoencoder task and machine translation task, are
proposed to be co-trained with ASR and ST tasks respectively. We demonstrate
that representing text input as phoneme sequences can reduce the difference
between speech and text inputs, and enhance the knowledge transfer from text
corpora to the speech to text tasks. Our experiments show that the proposed
method achieves a relative 10~15% word error rate reduction on the English
Librispeech task compared with our baseline, and improves the speech
translation quality on the MuST-C tasks by 3.6~9.2 BLEU.

本文提出了一种多任务学习框架，利用文本数据来提高自动语音识别和语音翻译的性能，其中包括基于注意力的序列到序列建模、去噪自动编码器、机器翻译等技术。在英语语音识别任务中，该方法相对于基线线下降了 10~15% 的词错误率，在 MuST-C 任务中有 3.6~9.2 BLEU 的提高。